Wife knows and supports, so no need to hide from her. However, we have a daughter (a very nosey one) and her family of 5 now living with us. Fortunately, I can keep most of my wardrobe in a second home several states away. We stay there often, and my wife calls it our "Girl Mecca."
At the main house, though, I've had to put a cipher lock on the MBR closet and to move some of the other stuff to a climate controlled storage facility. It has definitely thrown off my dressing rhythm at home, but family comes first.