That wasn't "her plan" - she was ordered to capture Rand and not to harm him.

She could not risk bringing more people and turn this to a battle, that multiplied the risks that it got out of hands and Rand got killed or harmed. If that happened, she was finished.

She couldn't get Rand not to bring a guard at all either, he would not have accepted. She tried to keep it small (but potent enough to reduce his suspicions and let him think with both genders included he held the advantage over the matching number of damane), and disguised as Tuon she obviously had hopes to reduce the bodyguard further for the "negotiations".

We don't know what her plan was once they were in the cabin but we know no one suspected she was a channeller (and no one would think the Seanchan would let a channeller unleashed...), and Rand's Aes Sedai would have told him she wasn't a channeller. No doubt after "Tuon" was seen to be an harmless little woman, she could have convinced Rand to negotiate in the cabin, with both sides leaving their escort outside, or to bring only one bodyguard each inside. The fact she could actually channel was a major ace up her sleeve. She could surprise Rand and capture him, and Travel away from the cabin leaving the baffled sul'dam/damane and Rand's escort behind. The domination band was another potential surprise. Rand's would not have suspected he could be captured this way, nor his Asha'man. Semirhage might have expected Rand had troops ready to jump into the fray, but they would arrive too late to do anything but fight the damane. Semirhage and Rand would have vanished already.

It wasn't a bad plan, considering the constraints Semirhage worked under, like Moridin's orders and the fact she couldn't be Semirhage openly with the Seanchan either, or really include them in her plans (they didn't know she planned to capture Rand). She couldn't, unlike Rand, have a small army of channellers of both genders waiting to Travel and fight at the first whiff of something wrong. Semirhage did a fair job at limiting the risks and isolating Rand, forcing him to hide most of his forces too far to interfere efficiently. It's not her competence that was her undoing, it's Cadsuane's ter'angreal. If anyone's to blame for what happened, it's Cyndane and Demandred who obviously didn't report to the others after the Cleansing that Rand had gizmos to break weaves. And perhaps if Moridin had done all this secretely but explained his plans at a meeting, Demandred would have brought up the danger Rand's people could collapse Semirhage's inverted mask of mirrors. It's not really Semirhage who is to blame - she was the victim of the Shadow's general flaws, of the way Shai'tan promotes competition and does everything so the Chosen can't work as a team. It's Shai'tan who doesn't understand he can't get the "best of the crop" hyper competitive deadly servants without this competion, their despise and hatred of each other, their mistrust, their egocentrism and their ambitions constantly preventing them from pooling their efforts as a super team. It's Shai'tan who lost the Shadow the advantage of the Chosen's competence. Together they would be frighteningly competent, but they cant work together.

As for her "stupidity" to torture Rand when she had the chance and despite orders, that's not really stupidity nor incompetence. She's a psychopath and a sadist who on top of it had been seriously humiliated by Rand's people - not a good cocktail. It's Shaidar Haran who underestimated Semirhage's pulsions and overestimated her reliability in such a situation. Any psychiatrist could have told SH Semirhage couldn't be trusted to handle this mission and he better keep an eye on it (which actually, it appears he did. Rand didn't get sudden access to the TP by sheer luck)