Umm, yes. Yes it is. At least for me, the only division I'm interested in (for the purposes of gendered spaces) is between women and not women, and most importantly men. Anybody whose legal identity is "F" needs to have consistent legal treatment as a woman. Anybody who doesn't, well, doesn't.
The debate over bathrooms should be over whether they should be gendered at all, not over whether men should be allowed into the ladies room.