It's not just the pseudo-adultery, it how he handled it when it came out. He didn't fess up, he didn't say "No Comment, my personal life is between me, my wife, and God", or even just lie without details "No, the allegation is false", instead he tried to blame his very real actions on right wing conspirators. He never apologized for slandering the right either. Doing it in the first place is immoral, doing it and getting caught and spinning a web of lies that smears others takes it into a very real and legitimate public concern.

Having an affair doesn't necessarily mean you're a callous bastard, doing it again when you've already gotten to watch it humiliate your spouse and while you're clearly revving up for a comeback to public life just says nothing good about the man. It only gets worse when you realize he must have known there was a very real chance that any young lady he was doing this with would be dumped into the public view. It still comes back to character though, because most of us want to see a strong capacity for self-restraint and grace under fire in our leaders, and he's proven he doesn't have those anymore than he has honesty or concern for how his actions might hurt those around him. People trying to wave this away as just sex and personal life, well I get that, but even if one thinks his actions didn't deserve public scrutiny, his actions once he was under public scrutiny did, and more, he should have anticipated the consequences of his actions if he was going to do them and planned better, and that in and of itself doesn't speak well as an endorsement to govern other people's lives. A good leader ought to be able to foresee crises and plan ahead, and failing that too handle them well. He clearly can't.

No honor, no foresight, no self-restraint, no good under pressure and no hint he really cares about how his actions made other people victims. So no business asking other people to trust him to govern.