I don't think inbreeding with mice is much of an issue, it is done all the time with lab mice and they are inbred for hundreds of generations to make them genetically identical. They still seem to breed and reproduce just fine.
I've been breeding mice from the same original stock for about 2 years now and they are all breeding just as well now, if not better, as when I got them.
You should have just 1 male with each colony, and once you put them together you really shouldn't move them again or you will have fighting and possibly lose some of them. Plus every time you move them around it will delay the breeding that much more as it takes time for them to get used to each other.
I've been breeding mice from the same original stock for about 2 years now and they are all breeding just as well now, if not better, as when I got them.
You should have just 1 male with each colony, and once you put them together you really shouldn't move them again or you will have fighting and possibly lose some of them. Plus every time you move them around it will delay the breeding that much more as it takes time for them to get used to each other.