It may well be considered an "addiction!" But if so, then YES you can let it go! First you have to admit it is there, and then detirmine to get rid of it.
But....in CW's case I believe he has a very low opinion of himself! Why, I have no clue. Except maybe for the way he was raised. I did not have a real happy childhood myself. I lost my mother at age 7, my dad at age 14, and was traised by an aunt and uncle who had two children of theirwn. Guess who came first in that family? But I did have the advantage of a wise Grandmother. She taught me three things which I have made lifelong philosophys to love by. They are: The Golden Rule (you should know that one), Always believe in yourself, and only worry about the things over which you can exert control.
In my 70 plus years on this earth I have experienced many things of great joy, and an almost equal number of tragic things. The worst of course was losing my wife of almost 50 years. But through it all, those three simple things have kept me going. And they can keep you, CW and any others who think like you, going in the same manner. Why worry about how others think of the way you dress? (with the exception of your wife or SO!) You cannot change their opinion, nor should you. Lead your own life, don't let someone else do it for you!
CW, your Tag Line says it very well! As does mine. I am a man who likes to dress! Nothing more, nothing less!