It is a little more than that. Blanchard defines AGP as being paraphilic, which describes "sexual arousal to objects or situations that are not part of normative stimulation and that can cause distress or serious problems for the paraphiliac or persons associated with him or her" (Paraphilia). An AGP would not be able to have sex with a partner without engaging somehow in the paraphilia. So the partner in this situation is an accessory rather than the (edit:objectsubect) of desire. This creates an empty sexual experience for the partner.
I agree with most of your points, but not this one. Women do not get sexually aroused over having breast implants or undergoing any other physical "improvement". They get excited about the sexual reaction they think these improvements will cause in a male.
The term describes a condition and it is not meant to be perjorative any more than other psychological terms. Blanchard theorized that non-homosexual TSs were all AGPs but this is not fact. You and others are proof of this since you are not AGP. And I venture to guess future studies will disprove Blanchard's theory. But it doesn't mean the condition doesn't exist. There are far too many people who say that it describes them to dismiss the term entirely. See "this thread".
There will never be a unified theory since there are far too many differences among the members of the TG community. People will need to become comfortable with the idea that certain conditions apply to some people, but not all.
You describe a paraphilia. Yes, a sexual fetish is also an arousal brought on by an object or a situation, but it is a disorder only if it is debilitating. It can also be an enhancement to a sex life if it is mutually enjoyed between partners. It is when the fetish replaces or becomes more enjoyable than the sexual relationship with a partner that it is a paraphilia.