I find it really hard to understand how your wife can demand that you grow a goatee and that you keep your chest hair. I think you need to put your foot down and say you are going to shave yourself where ever and when ever you want.
I had to assert my right to crossdress to my wife.