In my experience, neighbors do notice what goes on. The day I moved into my neighborhood, the lady across the street was standing in her front yard watching. Then she came over and introduced herself.
There's another house nearby where the teenage son has some shady looking friends and they visit the house. Several of the neighbors are keeping an eye on the house and calling the police when things don't look right.
Personally, I don't care what people I don't know think of me but it would bother me to have neighbors think or know that I was a cross dresser. And it would embarass my wife.