Why is it OK for you to dress up for role-play, or to express your feminine side, but not OK for others to crossdress for sexual enjoyment? If there is a perception that there is a sexual aspect to crossdressing, it is because there is (for some - not for all).
