My first thought on reading the OP’s post was a Japanese anime cartoon in which a crossderssing dad was elated that his son also wanted to be a crossdresser. The dad went on and on about how “only a man can truly cross-dress, because women already routinely wear male clothes!” He has a point. A woman wearing pants or pieces of male attire is so common that no one notices, in most countries. She has to sport facial hair or use a packer in her pants to present a male bulge down there, before anyone assumes she is trying to be seen as a male. Even cutting her hair in a short male style doesn’t get the same effect as a male wearing an obviously feminine blouse, skirt or wig. They just think she is lesbian if she severely crops her hair.
Among the transgender community, as opposed to cross dressers, it seems to me that the FtM trans men ‘pass’ much better than most MtF trans women do. In part this may be because hormone therapy works better for FtM than for MtF. But I think it is also because an effeminate male is still more likely to be seen and accepted as male than a woman with masculine traits is to be accepted as a woman. So unless they tell you, it is quite likely that you see far more trans males than you realize. Trans men are less visible.
I have met quite a few people in the CD, Trans and Drag Queen communities, in two very different geographical regions. This includes both social groups and therapy/support groups. MtF seems more common in all of them, by a 3 to 1 margin or better, when one excludes obvious or known lesbians who dress male while still presenting female. But this may just be because the FtM guys need less support, because they pass easier. I have talked to quite a few FtM trans men, and yes, they often have the same struggles that MtF women face. But once they cross the hurdle of coming out as trans, and get fairly well along in their transition, it seems easier for them to vanish back into society, comfortable in their new gender role.