Without seeing you this question can?t be answered without knowing more like do you have a full beard? Do you wear a skirt without any outher fem attire ? Do you have breasts or use something to give that look? Do you try to sound feminine when you speak?
If you or anyone go out as a guy in a dress or this case a skirt and are all male persons might identify you as a male and address you as such
If it bothers you then correct them when they say sir you say it?s mam or madam or women
Hopefully you can sort this out
