That isn't selfish at all, hon. My wife and I share many activities. We don't share all of them. That is perfectly fine. The two of you don't need to be "as one." At least not every minute. Perhaps it feels different because you are going through a big change as a couple. It is really not selfish to have a little time for yourself for something you love. Of course, if you aren't enjoying it, it's OK to set it aside for a time to, until you are ready for it again.