Well, as much as the GG's may cringe from my saying it, the truth is that older women often take on a few of the masculine physical traits.
Men are cursed with some things in our old age - gray hair, no hair, beer belly (even those that never drink a beer) etc.
Women often also pay a price for continuing to play the game of life. Often they will begin to grow facial hair, their voices may deepen or become a little rough, their hair may also thin, and countless other traits normally thought of as belonging to the the male of the species. While most people probably don't realize this on a conscious level (except the poor women affected this way) I think most people know this on some level and it therefore becomes a bit harder to tell an older woman from a cross dresser.