Really it's no different than being a classic car owner and wanting to go to the club rally. People with a specific interest tend to get together. The difference is that you can bore your non classic car owning friends by talking about your hobby but unless you're totally open about your CD'ing it's not something that your wife can reply to when a friend says, "So what did you do Saturday night? "
So yep I can understand your wife's stance.