Isn't that sort of what we're talking about? That someone of color is forced to be known simply as "black" or "African American" because he looks that way? And because a certain greater sympathy may exist for the group that was more willing to accept him?
He is biracial, because though he's felt the sting of this "shared history," many of his thoughts, reactions, personal historical notes, etc come from a white (for lack of a better term) background. And being of mixed race carries its own drama (some would argue that it can be worse than belonging solely to one group or the other), so how can you feel comfortable dismissing the actual life he's lived, and then tell me I'm "hand-waving away" anything?