I think it is a lot more simple than you may think. By "their" standards your behaviour is insane and unless you are a mental health specialist or educated on the subject, seeing someone act insane IS scary.
If your best friend, or Mother or whatever, started to count trees out loud every time they passed some, it would make you feel uncomfortable and alarmed and you'd suggest they stop doing it...maybe obliquely and helpfully, but you'd show disapproval.
It's exactly the same reastion when someone acts insane by dressing opposite to their born gender. It is scary and incomprehensible to most folk so it generates fear. And then anger for making them feel that way.
BTW I want to give a "Brilliant Post" medal to CaptLex.
(Anyone know what CaptLex looks like?)