Sorry, this got long-winded. I'm just not sure I understand the nuances here. Typical.
The average person is going to take "racist" to mean Something Bad that one individual or group does or says to another because of race. Do you stop in the middle of the conversation and correct them - tell them that if they are referring to an act against the powerful majority, it's only "racial prejudice?" I can't decide if this is fair, no matter how academically accurate it may be.
People have power individually, and I'm not sure that if someone hurts me because of my race, I wouldn't be able to call it racist behavior. Not just racial prejudice, which implies thoughts/feelings rather than actions.
This only comes up because of an argument I observed once - a girl of Asian descent was going after whitey, basically. She claimed that no matter what, whites cannot be victims of racism. That's fallacious anyway, because we aren't always a powerful majority, but I digress.