In my view, that's the consequence of trying to have a "male identity" (or a "female identity") in the first place.
A "male" (or "female") "identity" is nothing more than a selection of pieces of your self, edited and warped to fit some external standard of what a man/woman is supposed to be. If you split off enough pieces of yourself and label them "not me", they come back and, as you put it, wrestle for possession. But it's you who've set up the "winner take all" in the first place.
It's why I think that thinking of oneself as having a "male side" and a "female side" is unhealthy. They're both the same person (=you), and by splitting one's person(-ality) into two pieces, each of which claims it is you, you set yourself up for an internal power struggle. Not to mention a lifetime of misery.
Me, I have no use for "male identity" or "female identity." Instead, I'm striving to be aware of and understand all the pieces of my self and become a single, integrated person, instead of a collection of (warring) "identities."