That wasn't "her plan" - she was ordered to capture Rand and not to harm him.

She could not risk bringing more people and turn this to a battle, that multiplied the risks that it got out of hands and Rand got killed or harmed. If that happened, she was finished.

She couldn't get Rand not to bring a guard at all either, he would not have accepted. She tried to keep it small (but potent enough to reduce his suspicions and let him think with both genders included he held the advantage over the matching number of damane), and disguised as Tuon she obviously had hopes to reduce the bodyguard further for the "negotiations".

We don't know what her plan was once they were in the cabin but we know no one suspected she was a channeller (and no one would think the Seanchan would let a channeller unleashed...), and Rand's Aes Sedai would have told him she wasn't a channeller. No doubt after "Tuon" was seen to be an harmless little woman, she could have convinced Rand to negotiate in the cabin, with both sides leaving their escort outside, or to bring only one bodyguard each inside. The fact she could actually channel was a major ace up her sleeve. She could surprise Rand and capture him, and Travel away from the cabin leaving the baffled sul'dam/damane and Rand's escort behind. The domination band was another potential surprise. Rand's would not have suspected he could be captured this way, nor his Asha'man. Semirhage might have expected Rand had troops ready to jump into the fray, but they would arrive too late to do anything but fight the damane. Semirhage and Rand would have vanished already.

It wasn't a bad plan, considering the constraints Semirhage worked under, like Moridin's orders and the fact she couldn't be Semirhage openly with the Seanchan either, or really include them in her plans (they didn't know she planned to capture Rand). She couldn't, unlike Rand, have a small army of channellers of both genders waiting to Travel and fight at the first whiff of something wrong. Semirhage did a fair job at limiting the risks and isolating Rand, forcing him to hide most of his forces too far to interfere efficiently. It's not her competence that was her undoing, it's Cadsuane's ter'angreal. If anyone's to blame for what happened, it's Cyndane and Demandred who obviously didn't report to the others after the Cleansing that Rand had gizmos to break weaves. And perhaps if Moridin had done all this secretely but explained his plans at a meeting, Demandred would have brought up the danger Rand's people could collapse Semirhage's inverted mask of mirrors. It's not really Semirhage who is to blame - she was the victim of the Shadow's general flaws, of the way Shai'tan promotes competition and does everything so the Chosen can't work as a team. It's Shai'tan who doesn't understand he can't get the "best of the crop" hyper competitive deadly servants without this competion, their despise and hatred of each other, their mistrust, their egocentrism and their ambitions constantly preventing them from pooling their efforts as a super team. It's Shai'tan who lost the Shadow the advantage of the Chosen's competence. Together they would be frighteningly competent, but they cant work together.

As for her "stupidity" to torture Rand when she had the chance and despite orders, that's not really stupidity nor incompetence. She's a psychopath and a sadist who on top of it had been seriously humiliated by Rand's people - not a good cocktail. It's Shaidar Haran who underestimated Semirhage's pulsions and overestimated her reliability in such a situation. Any psychiatrist could have told SH Semirhage couldn't be trusted to handle this mission and he better keep an eye on it (which actually, it appears he did. Rand didn't get sudden access to the TP by sheer luck)

Moridin's plan wasn't bad either. Rand had opened his game: despite everything that happened with the Seanchan since TGH it wasn't enough, he still sought an alliance with them against the Shadow. Moridin didn't know his plan B was to go to war, had he known that he may have simply told Semirhage to be very antagonisitic and let events play out. But with Rand determined to make an alliance, he had to create a casus belli. His capture by "Tuon" would have resulted in war between AS/Asha'man, now leaderless, whose only hope to win the LB was now in captivity - and the Seanchan... soon to be divided into factions as soon as Suroth made a claim to the throne. Both sides would have lost its leader. The Shadow had to do something, it couldn't know or count on the fact even if it let Tuon and Rand meet, they would shoot in their own feet and Rand would mobilize while Tuon would attack the Tower...

I agree with you Sammael wasn't as such imcompetent. He too was the victim of the DO's plans. It's Moridin who saved Rand and lead to Sammael's demise, otherwise Sammael's (last resort) plans would have worked, despite Rand having trapped and destroyed his planned Shadowspawn army in SL by his nasty weave at the Waygate. Where Sammael made a terrible miscalculation is first to have agreed to Lanfear's plan. He should never have let himself become Rand's prime target, but mind you, his first mistake was to not have protected the secret of his alias better. Be'lal, Rahvin, Sammael - they all made the same mistake. It was a matter of time before Rand came for them.