... because you have a two party system. This is not the only issue where there is a multipolar set of opinions and views among Americans which get compressed into two competing, oftentimes extreme, views.
Specific to Affirmative Action, Asian Americans aren't (yet) threatening to abandon the Democratic Party over it. While you're right that there isn't as much support among them for it, nor is opposition deeply entrenched. So that view doesn't move the party all that much.
You can literally list thousands of such issues where some group has a different view, but it's gotten steamrolled by the two party machinery.