Personally, I don't care. I get in and get out.
However, I know many women who are strongly against/uncomfortable with Crossdressers, Transgender, or men using the women's restroom. Of those, some use the public space in the ladies room to change clothes for whatever reason.
Other friends were alarmed at a hotel on Halloween when crossdressers and men came into the ladies room. My friends were suspicious that it was just a ruse to enter the ladies room and very unhappy about it.
I also witnessed a man (on Halloween again), waltz to the front of a line of GG's waiting for their turn. It was brutal! The GG's read him the riot act.
I also have a close friend with strong views against it. She was alone in a public city owned multi stall restroom when a man (possibly a CDer) entered. She was alarmed and angry. She even reported him to the authorities. In her case, I would have been alarmed also.
GG's can't read the minds of CDers who show up when the ladies are alone. Keep in mind that GGs are concerned about their safety and feel vulnerable in restrooms. They do have the expectation that it is a safe space is for GG's.
GG's are the majority in their restrooms. Anyone who is not a GG is in the minority. If a lady has a hissy fit about a man in the ladies public restroom, the authorities will most likely side with the GG.
I also think that in these times, women are very nervous about who they see as men in the ladies restrooms.
This is probably not what you wanted to hear, but since you asked the question, this has been my experience.
Everyone is entitled to their own opinions