It's the "sept". Septa is a nun. So doing it in the nun is a whole other kind of perversion. As others have pointed out, it did happen in the book, and as for out of character, let's look at the other time the show portrays Jaime and Cersei having sex... oh, wait. We can't, because that was the scene that ended with his casual attempt at murdering a child. Among other murders alluded to and shown on the TV show, including that of a man established as a prior comrade in arms, just because he was pissed at the man's boss's wife.
Holding a knife to Ned's throat with glee, while his hiring slaughters all Ned's men? Heavily implied threats to Ros to get her to stop mourning a murdered baby (which only happened because Littlefinger helped put the killer in power) and get back to being sexually objectified? Actually having Joffrey torture Ros to death because she wasn't happy about serving a guy like that? Explicitly evil actions aside, did you miss his rant back during the sexposition scene where he's training the hookers to be fake lesbians, all about how he has been plotting revenge against Ned & Catelyn for years? Or how Ros warned Shae to keep Sansa away from him?
Too bad as a reader, you should know that Littlefinger is pretty much the arch villain of the series, and evil mastermind of just about all the bad stuff that has happened, as well as probably being the guy who convinced Joffrey to kill Ned.

You are seriously comparing the show FAVORABLY to the books in terms of dead weight? Which one introduced a completely gratuitous prostitute character, apparently solely for the purpose of titilation, and using up time that could be developing the consequential characters, only to kill her off without serving much purpose, other than possibly to illustrate Littlefinger's creepiness and evil. Said purpose apparently having been futile, as those characteristics have gone unnoticed. We get all kinds of dead weight like Lannister soldiers making crude jokes, in lieu of the actual battle of Oxcross. Rather than Oberyn's sexuality being casually referenced in a conversation, we have to have two more brothel scenes. The character in the books was a dangerous and unpredictable warrior, whose sexuality was merely the subject of rumors, which mostly served to illustrate who he considered himself unbound by cumstom or law. The guy on the TV show is a degenerate who lolls around in brothels with multiple prostitutes in two of the three episodes in which he has appeared, bullies lower-ranking men for being Lannisters, and is about as intimidating as Littlefinger. What could they have done with the money they wasted on sexpositioning the nature of his and Ellaria's relationship, which is really not that important to their characters? Emphasizing all the hookers they shared doesn't really add weight to her position as the voice of sanity when his daughters are spoiling for a fight over his non-murder.

If you are the type of person who needs to see movement to accept that something is happening, I guess the show is better fit for you. IMO, the stuff in Meereen was the most important, significant and interesting part of her entire arc, and did so much more to advance her character than a lot of wandering around letting people do things for her and tell her stories, in between bouts of indulging adolescent impulses, and getting away with it because dragons. In contrast to all the realistic interactions of characters and power or influence going on in Westeros, Daenerys story was a rather simplistic "return of the queen" type of story. Meereen is where she (maybe) learned that some problems can't be solved by mass murder, and the readers are signalled that her arrival on the shores of the Seven Kingdoms is not going to segue gracefully into "...happily ever after." It's where Martin gets realistic with the example of what is most likely to happen when you give even a well-meaning adolescent too much power (Robb being something of an abberration, who failed for lack of power, not a lack of maturity or restraint, and because Martin needed the traditional leader-types of House Stark out of the picture so the rest of the kids could rise from the ashes of absolute defeat).