I think that you've got it sussed. BMI is a statistical construct based upon a wide range of people and has little meaning for individuals. I'm 6'2" and 165 and felt that getting under 160 wasn't right for me. My father had a larger bone structure and would have been underweight at 165. If you have a small bone structure you might well be fine at 150. How do you feel and look? If bones are sticking out like a heroin chic model then you might have a problem, otherwise you're likely fine.
Statistically, men tend to have more bone mass and a greater muscle/fat ratio than women. Muscle is denser than fat, so men will weigh more than a woman of the same girth.