What does it matter what we think or what the rest of us do when it comes to facial hair in private? Your body (hair), your choice. If you're O.K. presenting as half man/half woman while still sporting that beloved 'stache of yours and are keeping it indoors, who are we to judge, comment, or even compare notes on the experience? What really counts is what your wife thinks of all this, and if she has no problem with it, then you kids go ahead and get crazy with it.
The only issue I would have with this is if you went out in pubic otherwise presenting as a woman, but still proudly showing off that lush facial hair. That would be a jarring visual for most "muggles" (yeah, me included) and would reflect badly on the rest of us as well - you know, the old "guilt by association" thingy. Said "muggles" would likely point, giggle and make rude remarks, and it would just reinforce in their minds the old "pervert", "weirdo", and "fetishistic transvestite" label that we've all struggled so long and hard to put to bed. To be honest, I'd rather not have deal with the prospect encountering that mindset when out in public en femme myself.