I will probably rustle a lot of feathers but here goes.
I get really tired of hearing that excuse.
Where I live you have rednecks and socialites, the poor and very wealthy and I really don't think they care either way how I dress.
Yeah I'm different to say the least and yet people just don't say anything 99% of the time.
So what I'm saying here is that is a lame excuse and to me it just shows ones lack of guts to just go for it and be yourself.
The only one holding you back is you.