I am not a woman nor am I a man... I am an enby. Hi, I am Wren.
Some fears are not in regards to physical harm although I don't want to diminish that at all. But for those who live in a conservative area, being "outed" can be just as or even more damaging. If you're a professional person or own your own business, you don't have the protection that "employees" have in regards to discrimination. And don't kid yourself... there are lots of "reasons" why a person can be dismissed that are unrelated to discrimination but can stand up in a court of law. Come to work late... poor performance reviews... the list goes on forever.
Then there's the whole thing about your standing in society... you volunteer on this board, serve on that committee... maybe even hold an elected office locally... all of that can be gone if you are found out. So there are a lot of reasons why someone needs to be more than a little careful when they venture outside of the privacy of their own castle. And they are very rational. You may get away with it in some ultra-liberal areas but I doubt if it would fly in many "Small Town, USA" locales.
Lacy PJs