I've yet to have an issue doing just that. Although I have qualms about doing it 5-6 generations in a row. Its really the best way to avoid introduction problems and ensure disease free mice in the future.
I guess I'm lucky in that my colonies are generally founded by 5 different males, and I can just mix 'n match males and females and get mostly unrelated bloodlines together.
After my extreme moment of dumb-assed-ness and got a new male from an unreliable petshop, that ended up carrying a lovely virulent strain of Myco, I lost quite a few of my adult female breeders. I'm just now rebounding 3-4 months later. *knocks furiously on wood* :headbang:
So I'm not going to be doing THAT anymore, I don't think. Most of the surviving ones have minor sniffles when it gets really warm or their cages are nearing time to be cleaned out. Their offspring are healthy and thriving. So I think I managed to get immunity going from that little stint and hopefully the last in the forseeable future.
But as contrived and contrary to textbooks as it sounds, its not that bad in the short run. I don't think there's enough of a problem with it because they live such short lives anyway. If you have issues with it in the future, just try to remember which female had which babies. Usually there's a minute size difference that you can tell, and keep back the healthiest looking from each mother. That will help to ensure at least some genetic diversity.