Societal norms do not currently allow men the freedom of clothing options that women enjoy. Ignoring sizing issues, there is absolutely nothing in my side of the closet that my wife couldn't wear to Walmart without getting funny looks. However if I were to wear a dress or skirt, or heels, or an obvious bra, or anything with spaghetti straps, or anything with lace, or even a t-shirt with the neck scooped too far or short sleeves that are a little too short, or shorts that are a little too short, people will notice. Even if they're polite and don't say anything, there will be thoughts and comments made behind my back (I hear from others when they see something like this). Even I notice on the rare occasion when I see a guy in something feminine (my second thought is envy, but sadly my first isn't). If I were to wear such things to a gathering of my own friends there would be much laughter and jokes (from experience here)
When I do hear comments I defend the man who had the courage to wear whatever, but my opinions only go so far in changing minds.
Cudos to those of you who, like John H, just don't care and own it. I wish I had your guts. I do push the envelope a little more when we're at our vacation home, where I'm not worried about business repercussions, but I'm still a long way from wearing a cami and mini skirt into a restaurant even there.
PS, I have several dozen women's t-shirts, and many more men's t-shirts. ONE of my women's t-shirts is uncomfortable fabric, only about 6 of my men's shirts are as soft as the majority of the women's shirts. Why??
