I think you have done enough. You do not need to sacrifice yourself to save him. If you're not into it you're not into it, tell him that before you build up a resentment. It's OK to have boundaries