I don't really care what male legislators think, but have you considered that the issue may not always be what the man in the ladies room is going to see, but rather what their presence means to the women inside? I have personally observed CDs (and various other types of self-identified "transfeminine" people) in women's restrooms behaving inappropriately, and frankly, in pretty discomforting ways typical of men. I was uncomfortable, and so were the other women in there. if a man was behaving that way on the street, I'd walk away as quickly as possible, and it doesn't get more comfortable when that's in my bathroom.
Editing to add/clarify... I know not all CDs/etc. would behave that way. I'm just pointing out that ENOUGH do such that the concern is not completely unfounded, and it's worth discussion.
I'm 110% for trans men and women who have legally/medically transitioned using the appropriate facilities for their gender, and having their legal gender marker respected. When it comes to situations where men can self-identify as a woman any time it's convenient for them... Not so much.
Non-binary typically refers to an identity, not a mode of dress. Which are you talking about?