I disagree.
That logic DOES apply to trans men and trans women - we are asking for (and at the federal level and in many states have received, at least from my perspective) an expanded definition for what legally constitutes being a "man" or a "woman".
In the case of non-binary people, part-time people, etc., it is a rather very different situation. You are not saying "we are women" or "we are men" and asking for the according rights. You are saying, rather specifically, "we are something else, but we want access to e.g. women's bathrooms too (in addition to the men's)."
There is a problem. The problem is that we have facilities that are (mostly) unnecessarily gender segregated along the binary. The correct solution is not to give you legal recognition as women whenever you're "dressed". The solution is to remove the gender segregation in cases where it's unnecessary.
When same sex marriage was legalized at the federal level, it wasn't because e.g. one man is now considered a woman for the purpose of becoming a wife. We just removed gender from it entirely, because it wasn't necessary.