Rand had used Callandor four times, two sucessful tries and two failures.

Successes: defeating Ishmael and destroying the Trolloc attack on Tear.
Failures: Trying to heal a dead girl and trying to destroy the Seanchan.

When he attacked Ishmael and the Trollocs, weaves he never knew about came naturally, e.g. pick out trollocs and fades and kill them whereever they were.
When he tried to heal the girl or destroy the Seanchan he struggled to figure out what to do and the results were unexpected.

Theory: Callandor was designed specifically as a weapon against the shadow. It's not just merely a Sa'angreal - it's actually designed to only work successfully against the DO.

Requiring women for other type of channeling is a safety feature - to ensure that caution is applied to using it (akin to having several people sign off before you can throw an atomic bomb).

So Rand will not have to have other women link with him to use Callandor if he fights the shadow with it.