I am in exactly your situation except that my wife is non-supporting of anything other than panties. She works from her office in town so I generally dress up, almost every day, and then go back to drab at 5 pm before she gets home. I could almost cry when I have to come down. I wish I could be me all the time. I also can't dress up if I go out. I personally could care less what others think but my wife is prominent in our small community and it would hurt her.
That said, my life has gone from dreading Mondays to wishing for Monday to get here...