My understanding is the "privilege" refers to an inherent advantage; it does not mean that everything came easy for you, just that it was easier for you than some other group of people, and because you never had to walk in those people's shoes, you don't even realize that you had an advantage.
So yes, I believe there is such a thing as male privilege. One study has particularly stood out for me. They had academics rate the CVs (academic resumes) of various fictitious job candidates. When they changed the name on the CV from male to female the candidate was consistently ranked lower. Interestingly, this was seen even when the evaluators themselves were women! So simply by having a male name, you receive an employment advantage.
