When I was growing up, kids used the word "gay" to refer to literally anything they thought wasn't cool. Those kids, for the most part, did not hate gay people. Should gay people have been okay with it because "they didn't mean it in a bad way"?
No. That is not how offense works. The same is true of men reducing women to their exterior/aesthetic, either explicitly or implicitly. You don't get a perpetual free pass just because you "didn't mean it like that".
