When I was in my late teens/early twenties, I worked at a movie theatre. Our uniform included some ugly double knit pants. I was talking to a cashier one day, I don't recall how we got on the subject, but she told me that it was funny when guys wore panties. She said that they think you can't tell, but you can.
I think she was trying to give me a hint as I often wore them to work. Aside from the bikini cut, I wonder if it was the rear crotch seam, like someone else mentioned, that gives it away. I always liked seeing that seam through the tight pants the girls wore at school. In fact, I like panty lines in general and lament that I don't see them very often anymore with the thongs and no-see-ums the girls wear today.
BTW, her comments didn't stop me from wearing them at work. Funny, the thought of a woman knowing what I'm wearing is fun, but I dread any guy finding out.