1:3 is fine in a 10 gallon. Just be prepared to clean them out more often. I found once I moved mine all to 20 gallon long tanks, the smell doesn't get too bad for over a week now. I'm currently running 3 - 20 gallon longs with 1:5, gotta ramp up for hatchling season ya know.
How many of one sex per 10 gallon? Well I wouldn't exceed 4 adults per any 10 gallon. The amount of ammonia in their urine is amazing, and if you crowd too many in without diligent cleaning you can have health and respiratory issues arise. You can keep younger mice at around 15-20 to a 10 gallon until they get fed off or euthanized for the freezer.
Never house adult males together for any period of time in any size enclosure. Fights will break out and infections can arise from potential wounds. They tend to aim for the testicles, so if they both get bit there, you're pretty much SOL.