My guess is that we are harder on ourselves than the general public is. The thing that we forget is that there is a GREAT variance in how women look. The scale goes from Ma Kettle to Halle Berry and everywhere in between and most of us will fall in there somewhere. Certainly there are some clues, such as hands, shoulders, etc. but usually there are things that we can do to deemphasize something or emphasize something else. It's not perfect, but it helps.
My wife and I were on an elevator with 3 women, and if you're on an elevator, you can't be that far away from anybody. From the side, one of the women gave me a complement and from the sound of it, I believed she was sincere. However, I did not turn to face her and limited my response to a quiet Thank You and a nod. But, had I turned or spoke with my regular voice, I'm pretty sure I would have been discovered outright. As it was, perhaps the information was inconclusive or at least enough to give the benefit of the doubt. A little distance helps, but I would bet that distance is closer than you think...