Sex is much more complicated than XX equaling female and XY equaling male. There are many people who are born with genitals that do not "match" their chromosomes, and they are not a rare exception. These people are intersex. Researchers estimate that as many as 1 in 100 people are intersex, so the condition is certainly not rare. Intersex people may not even realize that they are intersex; for instance, XX individuals may develop testes due to containing a fragment of the Y chromosome, but their XX dominance may make their external genitals and secondary sex characteristics appear entirely "female." Many people do not realize they are intersex until they seek help for infertility issues or other invasive medical procedures. In an operation on a hernia, one man, who was 70 years old and who had fathered four children, was found to have a womb.
Additionally, with the help of new technologies in DNA sequencing and cell biology, researchers are finding that almost everyone, to varying degrees, contains some cells that are sexed differently than the rest of their body. The assumption that all of a person's cells contain the same exact set of genes is false. So, even someone who has a penis and testes and mostly XY chromosomes may contain some cells with XX chromosomes.
Biologists are continually finding new evidence for the claim that sex is a spectrum that is mistakenly oversimplified into a binary. There is no reason to think that there are only two genders based on sex because there are actually more than two sexes.