I don't have direct quotes, but there were numerous references to the fact that the horn would only be linked to the person who blew it until they died. Turak said it in tGH, and in tDR Egwene and Nyneave were worried that Siuan would let Mat die so she could link it to whomever she chose. Then, the Aelfinn told Mat he would die and live again (I don't know if it referred to Rhuidean or Caemlyn, but those are the only two times he could have died). Mat mentioned a couple times that he was convinced he was dead in Rhuidean.

Who would be "in the know" about this? I suppose the horn Min saw when she first met the group could have disappeared, but would one missing item be picked out when they change all the time anyway? Did they even see each other between Baerlon and The Last Battle? From a literary standpoint, it's a common theme in tWoT that just about everything someone thinks they know for certain is wrong, even universally known facts (a Hero of the Horn can take over someone's body; Stilling/Gentling can't be Healed; insert your favorite example here). It became even more clear to me in AMoL, when they put so much emphasis on getting the horn to Mat, that there would be a "twist".