Do labels matter? It depends how meaningful your want your life to be and if you want to communicate with other people and have meaningful relationships with them. If you don't care about understanding yourself, communicating with other people, and having productive social relationships with them (that involve true love and such things) then continue to order the chicken when you want the waitress to bring you the fish. Keep in mind, also, that she will never love you because she isn't a waitress. She's a watermelon and you never tip her. The other women in your life won't love you either because you keep slicing them into pieces when you make a fruit salad.
Honestly, why does this question even get asked (you should ask yourself)? Gender labels matter because gender matters, to everyone. Misconceptions about gender cause a lot of grief in the world. You might have noticed it if you hadn't been so busy turning a blind eye to it.