I'll go from my experiences when I was younger and was in a customer service/service/hospitatlity etc. industry type job. On the first day of orientation it was beaten into our heads sideways to address everyone as sir and mam. On a typical day I could come in contact with anywhere from a few hundred people to thousands of people.
Every one of them was addressed as sir or mam, to where it becomes second nature. This was also reinforced by years of sports in which every coach was addressed as sir, all adults were addressed as sir. This has gone up to my current occupation where everyone is addressed as sir or mam, including superivors, managers, etc.
Even when at the check out line, resturuant, etc. I still address the staff as sir or mam. Basically it's one of the most common words I use on a daily basis. Now there are times when I am not sure of the gender, and when this happens it can sometimes be a double edged sword.
I'm sure I've probably offended a few GG's (who appear very masculine) by addressing them as sir, and vice versa by addressing very effeminate looking GM's by addressing them as mam . I base this off of their reaction when they angrily told me they or not a sir or a mam. Usually when I am in such situations now, I try to not use gender at all. Sometime due to being in a rush, not thinking, or letting my mouth speak ahead of my mind, I will still address them in the wrong gender.
I guess in trying to answer your question, there might be times where people are just being rude and trying to offend you. There might be other times when you will run into someone like me who just out of habbit and basicaly is like a robot programmed to say sir or mam every minute of there lives, we might occasionaly get it wrong.
That does not mean they are trying to offend you, and believe me on the other end it's not something I really like having happen and it does cause embarrassment on my end.
On a lighter note I get called mam a lot in male mode despite trying my hardest to look as masculine, "thugged out," and mannish as possible. When this happens I give a light hearted laugh, I'll actually smile for a change, see the humor in the situation, and just let it go.