That wasn't "her plan" - she was ordered to capture Rand and not to harm him.

She could not risk bringing more people and turn this to a battle, that multiplied the risks that it got out of hands and Rand got killed or harmed. If that happened, she was finished.

She couldn't get Rand not to bring a guard at all either, he would not have accepted. She tried to keep it small, and disguised as Tuon she obviously had hopes to reduce the bodyguard further while "negotiating".

We don't know what her plan was once they were in the cabin but we know no one suspected she was a channeller (and no one would think the Seanchan would let a channeller unleashed...), and Rand's Aes Sedai would have told him she wasn't a channeller. No doubt after "Tuon" was seen to be an harmless little woman, she could have convinced Rand to negotiate in the cabin, with both sidesd leaving the escort outside, or to bring only one bodyguard each into the cabin. The fact she could actually channel was a major ace up her sleeve. She could surprise Rand and capture him, and Travel away from the cabin leaving sul'dam and Rand's escort behind. The domination band was another potential surprise. Rand's would not have suspected he could be captured this way, nor his Asha'man. Semirhage might have expected Rand had troops ready to jump into the fray, but they would arrive too late to do anything.

It wasn't a bad plan, considering the constraints Semirhage worked under, like Moridin's orders and the fact she couldn't be Semirhage openly with the Seanchan either. She couldn't like Rand have a small army of channellers of both gender waiting to Travel and fight at the first whiff of something wrong. Semirhage did a fair job at limiting the risks and isolating Rand. It's not her competence that was her undoing, it's Cadsuane's ter'angreal. If anyone's to blame for what happened, it's Cyndane and Demandred who obviously didn't report to the others after the Cleansing that Rand had gizmos to break weaves. And perhaps if Moridin had done all this secretely but told his plans at a meeting, Demandred would have brought up the danger Rand's people could collapse Semirhage's inverted mask of mirrors. It's not really Semirhage who is to blame.

As for her "stupidity" to torture Rand when she had the chance and despite orders, that's not really stupidity nor incompetence. She's a psychopath and a sadist who on top of it had been seriously humiliated by Rand's people - not a good cocktail. It's Shaidar Haran who underestimated Semirhage's pulsions and overestimated her reliability in such a situation. Any psychiatrist could have told SH Semirhage couldn't be trusted to handle this mission and he better keep an eye on it (which actually, it appears he did. Rand didn't get sudden access to the TP by sheer luck)