Strangely enough, the ONLY time I've gotten any kind of negative reaction has been on Halloween, when you'd think people would be the most willing to let things pass (pun intended).
I'd always thought it had to do with the fact that people feel more free to say whatever comes to mind on Halloween, but on other days are a bit more conditioned to 'live and let live'.
Even then, the worst thing that was ever actually SAID to me at ANY time was when I heard 'Nice Tits!' yelled from the window of a passing car after I had walked in front of it at a crosswalk.
I decided to take it as a compliment.