... because you have a two party system. This is not the only issue where there is a multipolar set of opinions and views among Americans which get compressed into two competing, oftentimes extreme, views.
Specific to Affirmative Action, Asian Americans aren't (yet) threatening to abandon the Democratic Party over it. While you're right that there isn't as much support among them for it, nor is opposition deeply entrenched. So that view doesn't move the party all that much.
You can literally list thousands of such issues where some group has a different view, but it's gotten steamrolled by the two party machinery.
Yes this is the wickedness of the two party system.
Simultaneously.
No single vote can change the outcome of the two party system. The closest in recent memory is the election of Virginia in 2017 between Democrat Shelly Simonds and Republican David Yancey. Except you were not voting for the entire VA legislature, you were electing a single member out of a 100, it is just serendipity that the 2017 results besides this 1 seat was 50-49 and that 1 seat determined by random lot made it 51 R-49 D
(since that 2 year term, the new results after the 2019 election is 55 D-45 R.)
...
Since one vote does not cause the change by its own, only a sea of 1 votes aggregated together, from a different analytical point of view (not mine, but it is valid) is that it is perfectly valid to vote on sentiment, on your own values, even if the person you are voting for does not reflect those valids.
Why does one need to choose, between two choices that do not reflect themselves? Why does one need to invalidate the self and surrender individual agency?
...
Why then do I bring a point of view that is not my own? Well it reminds me as an individual to unclench. No matter what we do our system of legitimacy is beyond 1 person, thus burning relationships may not be the right action. At the same time I am all for burning relationship with people who do not treat others with respect* (and what is respect in my heart / mind is my own internal scale I will not share with others, for words can not capture it. It exists but I can not share it with words, just like I can't share the happiest experiences of my life, nor the most painful ones full of betrayal, the best I can share is a simulcra of these things which is not the real thing.)