For me, I tend not to feel guilty for dressing in feminine items. My overall emotions from dressing are positive. But, there are negative emotions involved because of reactions from my wife. I tend not to feel bad that I have the taste I have, but I do tend to feel bad that she's "stuck" with someone with my tastes. It's not that she doesn't receive benefits from being married to me, but I'm always aware of how ashamed she'd be for anyone else to know. One could make an argument that there's no victim, but the social backlash in my particular social circle could be severe for my wife and for me.
I started dressing in my mid-fifties and included her early. I left her a lot of control in the beginning and that seemed to work in maintaining a sense of safety for her. Since it's all experimental for me, I tend to try various ideas before allowing her to see anything new. Someone could argue that I'm lying by omission by not expressing every experiment openly, but I disagree. So, for me it's not about lying. On the other hand, I do feel inclined to hide certain things about myself but feel that she could take as much blame for making me hide aspects that I'd love to share with her. She's made sharing unsafe to some degree. Maybe her reasons are legitimate. But, I don't think that they're more legitimate than mine.
I don't think that it's necessarily breaking a natural or social law that leads to a feeling of guilt. I think that it's the awareness of the likely reactions of others that brings about negative feelings that we interpret as guilt. I cannot think of anyone making a legitimate claim to a "natural" law against wearing any particular item of clothing. "Men's" and "Women's" clothes are a social and marketing construct and even crossdressers support that construct by wanting to wear "women's" clothes. I know that I am guilty of preferring the same item from the women's department rather than the men's.
As I've said above, it's not guilt that I feel, but there are definite negative emotions associated with the legitimate feelings of rejection that come with crossdressing in intimate or social interactions.