As with most things, a certain set of people will assume, incorrectly, that being a CDer means something else. And I do think that societal attitudes are a reflection of the selfish genes that strive for replication. (We're little different from bacteria in that regard.) As a successful procreator (and grand procreator...if that's what you are when you have grand kids)...I know that being a CD doesn't necessarily present an obstacle to reproduction.
Also, not every species demands successful reproduction from every adult. Look at any number of social animals, from ants to wolves. You'll see that some members of the community abstain from reproduction in order to enhance the succesfful survival of others...usually their siblings. The gene gets what it wants ---viable offspring--- and males of the species play specialized roles, sometimes not typically associated with reproductive males. Think of submissive male wolves who hunt and care for the offspring of the Alpha couple.