Red herring. Sorry but I believe this is an ignorant and insensitive attitude. Maybe if you lived with the experience of a sexual assault you might understand that anxiety and fear come from the past, regardless of whether it is likely that an repeat attack will occur, which it rarely will. A woman who was raped or attacked, and unfortunately there are too many, can experience significant negative feelings if a man appears unexpectedly in her space - and many women do believe a women's rest room is their space.
It is funny that we feel so strong on this site about wanting respect and dignity for ourselves, yet we don't try to understand the feelings of people we may truly scare by our appearance. We do not feel we threaten others but we can and we do, just as we often feel threatened and disrespected. What's the difference?
Maybe a touch of empathy will promote greater mutual respect and understand.