I have been shaving legs, arms, and underarms for some time now and I wear shorts and tee shirts everyday at work, and some times clear nail polish. The one thing that I have noticed is that IF anyone has noticed they have never commented. I see lots of guys at the beach with no hair on their bodies and as the others have said lots of swimers and bike riders shave to reduce drag.
So its up to you, I say go for it, but fro my experience it seems to be no big deal.
Hugs Phyliss