You may want to ask him what finally caused the failure of his first two marriages. I can see where the ground rules that were established have been slowly chipped away. It seems if you give him an inch, he takes a mile. I can see that you are thinking, "OK, if I give him some more latitude, that's will satisfy him." Frankly, I personally do not think a successful marriage can exist, if one party keeps pushing his or her own agenda at the peril of the other. As a cross dresser married to an non accepting wife, I would not wave my panties in her face. You are justified in asking to revisit your acquiescence on the issues because you realize it is NOT working for YOU.
PS: Don't wait until his retirement to address these issues!