I just finished reading Feinberg's Transgender Warriors and there was a paragraph or two about passing and how we feel the need to pass because of fear of harassment essentially.
I think for me it's part of it but it's far from the main reason.
For me, the ability to pass means I'm successfully presenting as male. I try to pass so I will not be seen or read as female. And I try to push it so that I will not be seen as gender neutral as I want to be read as male because really, I am male. Despite what's in my boxers currently.