It could be neither. Different strokes for different folks! No pun intended or maybe so. Anyway, I think if a young boy had a tendency to wear women clothing or at least panties there would be some sexual component to it. Not having ready access to willing females generally leaves little choice. You have to take what's at hand. Masturbation among adult men is prevalent so I do not think wearing women's clothing may be an enhancer rather than a substitute for access to willing females.
I dabbled in my mother's lingerie draw because I was drawn to nylon. Slips and nightgowns were the entry way for me. I pretty much functioned without any assist from any female clothing. I never had the thought the clothes made me sexy. I would not characterize how I feel looking into the mirror over the years. I think attractive would be a more suitable term. I had no problem attracting the attention of young women in my youth or early adult life. Or, as my wife would also attest as she fought off gay men, it seems she noticed more than I certain men sought my six foot two, 175 pound, former military physique. I've always felt a lot better about my male physical image than the female reflection in the mirror. Although, I still enjoy the feel of those nylon slips and panties.
And, you should not feel shame or view yourself as perverted. I will offer, if anyone is getting more satisfaction from dressing up than getting it on with their wife, you may have issues in your marriage.
