Moiraine's balefire seems to have been enough to destroy Bel'al before Shai'tan could intercede thus neither being partially over a gateway nor being partially in the Dreamworld are necessary factors to prevent transmigration.

My money is on the idea that the balefire was woven using so little Power that it only shaved a few seconds off of Asmodean's life, which means his soul may have been burned backwards and re-inserted into the reincarnation pool before it could be snatched by the Dark One in a coherent form.