We traditionally say "rights" but marriage is, from a legal perspective, a privileged relationship in which the partners enjoy certain legal preferences that are not granted outside marriage. I think this is the key point that must be stressed, however, because the straw man argument that gay marriage will lead to polygamy, letting people marry animals, etc., is a flawed one. If we say that we have created a special legal status between two individuals, then same-sex couples have a claim that the way the privilege is set up is discriminatory. This is an equal protection claim. Imagine if marriage were defined as "the legal union between one white man and one white woman" and you see the crux of the argument. The "rights" only come in when we look at the Constitutional right to equal protection under the law.

If, however, the privilege is set up as one for only two people, then people who say they have a polygamous relationship are not in a position to claim that they do not enjoy equal protection, because the relationship has been defined as a two-person relationship. After all, if we expand to three, we could expand to 100, or 10,000. So, consequently, we can create a privilege for two and exclude three (though we could just as easily allow it if we wanted to, without being obligated to on a Constitutional basis), but I don't think we can create a privilege for two and then set conditions on who the two people can be.