Of course one can be and stay away from trans-rights issues. Although I think we have stretched the definition of transgender a bit too far.
I'm not judging anyone, but if a person only wears panties (or stockings, or any other garment) under their guy clothes, has no desire to either emulate or be a woman, feels 100% comfortable with being a guy, and proceeds no further in crossdressing, would they be as concerned with trans-rights as someone who is living 24/7?
If you've (general pronoun) ever been out in public, and had concerns about which bathroom you should use, and the reaction you're going to get, then you need to be concerned about trans-rights. I mean, after all, aren't "Men's Room" and "Women's Room" simply adjectives too?