Many of us are consumed with the illusion of passing, of hopefully, one day, being embraced as the women we are or want to be. They work really really hard at trying to "pass" and usually have all kinds of hangups over any masculine traits that make this endeavor harder than they'd like. That kind of person is typically repulsed by the idea of a MIAD or a bearded crossdresser. You just don't fit into their ideal.
However, among the younger generation, and particularly in more coastal, progressive places, the MIAD approach has gained great popularity. I often seen twenty-something guys with their lumbersexual beards casually sporting dresses, without a care in the world.
While I must confess that this does not fit neatly into my ideals, I think it's wonderful that society can see a guy with a beard in a dress and not melt down. That's called progress.
So keep doing you and don't listen to anyone. I'm sure you look fantastic.