I have not been challenged since my first going out some 40 years ago.
After my SRS and going out with men, some things are always in your mind. Some men don't like being told NO. After some years I was worried that someday someone would have questions. TS was rare in those days, but that was NO guarantee of safety.
But that is not to say that being a woman out and alone is not to be safe either. Women do get harassed and assaulted and raped and even killed sometimes.
One should always be aware of their surroundings and situations.