Remember Elaida lifting Rand's sleeve and saying his untanned skin being so pale made it questionable he was from there? And Rahvin, who is very dark skinned, was able to pass as a Lord from Western Andor, without it eliciting any comment from Mat when he saw him.
That said, Mat being substantially more fair skinned than the other three original TR folks is obviously not going to work, but they did say they were going to do a fairly colorblind casting, and they seem to have held up to that.