actually in the theory section and everything, on this topic.

My idea was that a person's soul, their thread in the pattern, is continuous. The Taint, caused by the Dark One, is corrosive to things that are "natural" (ie: made by the pattern). As such, when a channeling soul is reincarnated into an era when saidin is tainted, their soul is constantly in contact with the corrosive taint, which eats away at the natural boundary between their reincarnations.

Because of a variety of reasons, it is extremely unlikely that a person could live long enough to begin hearing multiple voices from their past lives. For one, the Taint also corrodes the physical body of the channeler. Secondly, the society that the person was raised in teaches them that being a male channeler is perhaps the most evil, horrible thing possible, causing them to loathe and fear themselves, so they are unlikely to channel much more than absolutely necessary. That lack of channeling will keep them apart from the taint, for a while, until it gets to be too much.