Stephanie,
You have posed a wonderful question and it can lead in many directions.
What other types of behaviors can be considered selfish? Forget about golf and hunting---what about shopping? or talking on the phone? What about getting dressed for a party? My SO loves to get dressed for a party. She generally spends about 1 - 1 1/2 hours getting ready and she really loves it. Is she guilty of being selfish? She could probably get ready in half the time.
I happen to agree with Rhonda Jean---people choose to call CDing selfish because they don't think it is "normal". Shopping is normal so you can spend hours and hours shopping and it is ok.