The fact that you refer to females as "the weaker sex" partially answers your own question. Historically, women have been marginalized and devalued. That a man would want to drop down to that level of lesser privilege makes him weak in the eyes of those who view women as less than men. So that's part of the problem.
There's also the problem of society viewing gender as binary and the burdensome roles assigned to each gender. It isn't binary, and these roles are not always useful.
Intellectually weak people also tend to fear what they don't understand. Suddenly someone as innocuous as a man in a dress is considered a threat. A threat how? Same way gay marriage is somehow a threat to "traditional" marriage... it's different.
And I'm not a psychologist, but I do have a degree in psychology. All depends on one's area of focus/research as far as their expertise. I was focused on a specific type of PTS(D), but I never went for a doctorate.
