I think many people are still attracted to a gender binary
So, as women's fashion has changed, men's fashion has changed to compensate and support the binary. For example, as short shorts have became very popular on women, they are now viewed as a faux pas on men. When this happens and the binary is well suppported, we still have ample visual clues whether someone is male or female. Many people want that to be something that can easily be determined and are uncomfortable with living in a world where it is difficult to identify the gender of a person quickly and easily. Ever since razors were invented and men could shave their beards (the most emasculating grooming practice there is), mankind has been confronted with finding a way to determine gender via adornment, clothing, grooming practices, etc.
However, there is a counter-cultural movement against the gender binary and we are all members of this movement; some more visibly than others.
I don't think this explains everything but I think it is part of the equation.
Jane Boys Are Gender Pluralists
Most other people don’t believe that TG people have positive social value. In one common argument, TG people are viewed as having negative value because we are perceived to have negated our natural sex and gender. We are perceived as people who have an artificial sex and artificial gender. Our artificial sex and gender is deemed to be inferior to the natural sex and gender of other people.
Furthermore, artificial femaleness and artificial femininity are viewed as the lowest forms of any possible sex and gender expression. This is the case because we are prone to believe there is one best form of everything including sex and gender. The absurdity of this belief does not register with a large number of people. Consequently, males are widely ranked as superior to females, and masculinity is widely ranked as superior to femininity.
Life will improve for everyone when social majorities stop believing in gender monism (one uniform superior gender) or gender dualism (two uniform superior genders) and start respecting gender pluralism (diverse types of each gender).