In pre-industrial society population growth wasn't a given. Sometimes it expanded, certainly, but other times it went the other way. Between c. 1200 and 1400 the population of England roughly halved - part of this was due to the Black Death but that's not the only cause: if it was there'd have been a big dip followed by a rebound. Actually even when there wasn't a plague there was a small but steady decline caused by a combination of lesser diseases, bad harvests (some good of course, but more bad than good and so on). During this time many areas reverted to wilderness, forest and similar.

A similar event would only have had to happen for the last 200 - 300 years of the New Era in WoT to produce a similar effect to what we see in the books. It could have been entirely natural (there was no Dark One in Mediaeval England!) but I also like Datakim's idea that it could have been some very slow and subtle seepage from the gradual weakening of the seals.