I agree. It is not like you are doing something that is illegal or harmful. Why quit? You are not doing anything that is illegal and it is not hurting anyone.
There is nothing illogical about the comment. We do hurt others by quitting. Many CDs quit and then have mood swings, depression, and other mental problems. Those problems are real and affect other people.