I'm not one to throw the word "fetish" around much. The need to redefine the word in order to "be justified" defeats the idea of "definition" in the first place. I can "feel justified" to crossdress, but justification is like beauty. It's in the eye of the justifier. I can justify my behavior any way I want and can be denied justification just as easily by anyone else. From my point of view, I'm not trying to get justification. Justification will only lead to tolerance at best. What I'd be interested in is approval and affirmation.
It seems like a circular argument to say that crossdressing, 'fetishistic' or otherwise, is the result of being deprived of being able to crossdress.
I was a latecomer to the idea of wearing clothes that are typically reserved for women. Before I ever considered the idea of wearing panties, the thought never occurred to me that I was deprived. Wearing them, at first, did at times precipitate some 'fetishistic' behavior, but it wasn't deprivation that caused it. If anything, it was the novelty of a new experience.
What you might be describing is more along the lines of Maslow's Hierarchy of Needs where it's stated that "once a need is met, it no longer motivates". That would speak to the idea that those who's need for perceived acceptance in an area are less inclined to be stuck trying to get that need met.