As an experiment I put on a pair of Levi's and a t-shirt to watch an episode of The Big Bang Theory. I then watched an episode wearing Gloria Vanderbilt jeans and a woman's Apt 9 t-shirt. In both instances I found the women to be appealing and felt no attraction to the men.

Taking it one step further, I got fully dressed (pantyhose, bra, forms, skirt and blouse) but the results were the same. Maybe I am just wired differently.