I agree with you insomuch as gay people have their own agenda and beliefs.
They also differ from ours, we still like women where they are attracted to people of the same sex.
We like clothing of the opposite sex and this gives us a drive.
Men do have many fantasies about other men and and these are quite different to cross dressers.
Women also have a totally different attraction to other women and their fantasies can include men or even to the extent of abhorrence.

The only tangible link I see is that we have all been attracted away from what people see as a "normal" hetrosexual relationship.
We are driven together because society defines us as different.