Don't worry, it's not usually intended as hostile. It's more of an acknowledgment that you are there, almost like a greeting; and at the same time a way of sizing you up... to make a quick judgment about whether you might be a friend or foe. It really differs from city to city though. In friendlier places people will look at you more on the street.
I once read a study that said women tend to look around a lot more than men, and men don't understand the behavior. It's just part of the subtle differences between us and how our minds work.
...or maybe they're just checking you out because you look good. I did that to a very pretty crossdresser on the bus the other day. Hope I didn't freak her out. That wasn't you, was it?
Edit: Stephenie is correct. Smile.
