I have been crossdressing for over 60 years. When I was old enough to use makeup I started going out in public as Stephanie! Then when I got married, my late wife did my makeup and really did make me look feminine. I could pass quite easily!
When she passed away 7 years ago, I decided to try going out dressed enfemme but looking like the man that I am! The only comments that I have heard are questions as to where I bought the dress, skirt, or top that I am wearing! As long as you are decently covered, most people don't care what you are wearing!! If you do go for show. then yes you are going to hear comments, some of which will not be good! But that is your own fault!!
As has been said, most women who dress in what seems to be mens clothing are usually wearing man styled clothing made for a woman!! And I doubt that many women wear mens underwear!! I am a man, but I wear panties and a bra almost every day because I am a crossdresser!