The whole genre is the child of ancient heroic or mythological epics and though there's often more spiritual layers involved in the growth of the hero(es), it's difficult to imagine it without a martial aspect, or at least physical fights or deeds involving violence.

For the rest, it depends a great deal on how you define "fantasy". Personally I frankly don't see the point of using the "fantasy" label for works like Marquez's, or Dickens's "A Christmas Caroll", or Alice in Wonderland (though that one and Peter Pan are debatable as they are very close in spirit to modern fantasy), or some novels by Murakami, Twilight, let alone old works like Rabelais, the utopian novels, Foucault's Pendulum, 1984 or Brave New World or the Greek Myths. It just renders the label "fantasy" ridiculously meaningless. I understand perfectly there are historical connections and filiations and that Tolkien's works for e.g. are created mythological (Silmarillion) and heroic (LOTR) cycles (while The Hobbit is just a modern fairy tale for children), but they have none of the functions real mythological and heroic epics had, or utopian novels or allegories etc. and it's the crucial difference. IMO it's one thing to acknowledge those things, acknowledge the fact that literature has a long history of describing/using realities beyond the mundane world, but Fantasy remains a genre that's essentially born in the 20th century, and quite different from its forerunners because it divorced the creation of secondary world or fantastical elements from the varied functions such literature used to have in the past (religious, philosophical, political, social, satiric, allegoric and so on).

The genre has already evolved and diversified a lot since Tolkien. That tradition survives mostly in epic fantasy. For instance, Tolkien never aimed for "realism". His world building was restrained to everything needed for an epic (or for his mythological cycle) and that's it. Only the hobbit culture was developped beyond that, to create a contrast wih the rest. His world is so detailed it appears almost alive, but what's brought to life in his work is a mythical world.

Later on, Fantasy writers started borrowing elements from science-fiction (the most influential writer for this is no doubt Frank Herbert and his Dune). Those didn't want to create something that would function as a mythical world (not even as elaborate/internally logical as Tolkien's), they aimed to create "realistic" worlds. They started caring about stuff like having enough farmers to feed everyone, elaborating diplomatic and political relationships, to distribute resources and have economies that make some sense, to look at anthropology and history rather than mythology to create races and cultures, to incorporate science into the mix and to find rationales for alternative technological development. It's Fantasy, but with a good dose of speculative fiction thrown in the creation of the world itself (where for Tolkien, it was still more a matter of reflecting mythologic/cosmologic themes and motifs correctly). The "degree" of realism, and the style the story itself is told in, vary a lot from author to author. You got people like Jordan who exagerrated traits for various purposes (humour/satire, or to create archetypal types of personality ), and you get people like Martin who decided he would go to the level of hyper-realism in the depiction of a "medieval" society - or at least their mores, with all the violence and scatological details it involves, or Erikson who made a lot of efforts to give a realistic picture of military life. It's utterly impossible to write a Fantasy story like WOT, ASOIAF, Malazan, Memory, Sorrow and Thorn and even Harry Potter and so on without violence. Remove war, make no character a warrior or warrior-type and those stories simply stop being what they are. At best, you can limit the realism of the violent aspects, avoid gore or graphic descriptions and so on, or restrict yourself to using those when it's necessary dramatically (in the Jordan vein, or even more Rowling), or you can use a much higher degree of realism if you want to convey how brutal, barbaric or unforgiving to the physically weaks your universe is. You can even choose to revel in the description of graphic violence, or sexuality, as a way to entertain your audience - then it becomes exploitative, in the vein of pornography, gore or graphic horror. It's what people who don't like the violence, scatology or sex in Martin have against his writing, by the way. They perceive it as exploitative or skirting with exploitation, there not enough to shock the reader with the barbarity or crudeness of life in his universe to make dramatic points, but as a kind of "fan service", the modern version of gladiator fights, or erotica/pornography. Personally I'm divided on the issue concerning Martin (though this aspect still don't really bother me, the sex and crude language certainly not, the scatology I can bear and while battles and fighting in general bore me, Martin like Erikson remain quite within my level of tolerance). My feeling is that most of it is there either for dramatic purposes or for the sake of giving a realistic picture of the kind of society his universe is set in, but I highly suspect there's some exploitation here and there, because he likes it and he knows his audience does too, so he indulges himself, and his readers. It's more obvious, though, from the "RPG generation" of Fantasy writers, especially with those who started crafting their world while still teenagers or late teenagers. They've kept a kind of juvenile fondness for graphic sex, edgy stuff and gore. I don't really have a problem with this, it's part of human nature and it's a huge, huge step forward to enjoy this by procuration in books or animation instead of exploiting other human beings to get the "real deal" on screen like with porn, or much worse, to act it out.

To get back to violence and fantasy, finally there are those who've got rid altogether of the "epic" angle. They still write Fantasy, but it's no longer so closely rooted in epics or mythology (or when it is, it's no longer attached much to the heroic aspects of mythology), so in theory they no longer need heroic characters, warriors or warfare at all. For all sort of motives, these writers still choose to write their stories in secondary worlds but very often, those novels are hybrids, cross genre. They are urban stories, or romances, or psychological stories, skirting with SF themes, or have intellectual or philosophical themes, being almost completely political or could even explore economy, or geographic research/discovery, or are mysteries/crime fiction and so on. It's perfectly possible to write novels in this vein with a bare minimum of violence or no violence, or at least to make graphic violence completely unecessary dramatically.

To touch on something else to conclude, I totally disagree with one of your earlier comment where you include "greyness" as a characteristic of "gritty fantasy". It's a completely different issue. At some point, this has a massive deal to do with the background of the writer. It was inconceivable for Tolkien as a catholic from his walk of life to create a Fantasy that would go against his Christian values. He created a Devil, and its through his agency that evil enters the story, and this evil is shown as exploiting the weaknesses, moral or emotional, of the characters. So yes, no "hero" character is by nature evil, but through despair, moral weakness, various forms of greed, power lust and so on, several characters became susceptible to exploitation by the "devil", from Boromir to Saruman to Wormtongue, Theoden, and even Frodo. Even Sam and Faramir would have killed Gollum (a sin), and though by the time of LOTR they're pretty wise and quiet, the Elves were hardly angels before. To Tolkien, this was what "evil" is all about.

Jordan rather made harmony good, and chaos evil - from the standpoint of humans, true evil is the end of the cycle of life or serving those who could end it, but the cosmologic role of the DO remains for now unexplained (it's hinted that he is the equal and necessary counterpart of the Creator, unlike Tolkien's "Devil" who was created by God like Lucifer) . The Devil is no longer responsible for creating Evil, his influence merely exploits "human evil" and pushes it to greater extremes.

Martin (so far) depicts a world where religion has stopped playing a very great role in defining the moral codes, especially among the ruling classes, so they let their human nature run the show, and often enough it's their basic instincts and self-interests that drive them. Most of them see their religion as an hinderance or something to ignore, and those who stick to the moral teachings of their belief system (like Ned Stark) thus get mowed down by those who let their self-interests be their moral code, like Cersei et al. It's not "gritty" or "realistic" in essence, it's merely a feature of his world that religion doesn't have the kind of power to hold most of the powerful in check, though if you notice, in recent books religion is making a return with a vengeance and its moral/ethics is attempting to define characters like Cercei as evil. This feature of the world building means that Martin himself takes a minimal role as arbiter of ethics/morality, and thus the reader is left to judge if he's so inclined, following his own beliefs and ethics. There's as much greyness or black and white as the reader himself puts in. If you apply real world ethical systems to ASOIAF, it's not quite so "grey".

There's also the whole Fire and Ice "war" coming, and we don't know if Martin won't draw a line somewhere (the increase in power of religious cults might hint at that), forcing his characters to define which side is "good" and which side is "evil" (which would bring things to a similar standpoint as WOT. Evil with a capital E would de facto be the side striving to destroy humanity, or those refusing to serve on the side trying to stop that). I think what you mistake for "moral greyness" in Martin is actually that like with his depiction of violence, he seeks to give construct psychologically realistic characters, so the characters justify their actions. That she seeks to protect her children or had a terrible childhood doesn't make Cercei's decision anymore moral or ethical, if you ask me, or that Jon made bad decisions and mistakes doesn't stop him from being an essentially very moral/noble figure.

So ethical or moral greyness isn't part of what defines "gritty". You can write morally ambiguous characters without any grit, and you could write a completely black and white fantasy that would be very gritty. Martin chose to include realistic justifications as part of his "realism", another writer could write "gritty" without any of that. It's more a matter of the writer taking a stance or not, defining evil or not in his world through creating ethics and religion, or evacuating them and letting the readers take side according to their own. And "black and white" do exist. Just ask an islamic fanatic about America, you'll be thrown right into a story of conflict between Good and Evil with capitals...

"Gritty" is merely a style of realism in storytelling that doesn't shy away from aspects such as bodily functions, gore, edgy sexuality (often including perversions or violence, such as rape and so on). When it's also "morally grey", it's often a device not to bound the characters to an ethical system by not clearly defining the world's ethics. As I said, it's not Martin's case. If anything, his personal stance in ASOIAF appears to be that religious fanaticism is bad news for a society, and more importantly that rare are the powerful individuals with the strength to define and abide by a personal ethical system when society doesn't share a strong one anymore, reinforced by both religious and secular powers. The religions on Westeros no doubt condemn murder and rape on a moral basis (the society is way too primitive for more developped abstract ethics), but few characters still care about religion, so there's only the risk of punishment to stop them from giving way to animal instincts. They won't rape if their lord will hang them for it and they risk getting caught. Otherwise... soldiers and killers rape "enemy" women and children, or did a lot. It's what happens when you deshumanize "the enemy" and there's no (or too weak) legal, penal or moral/religious safeguards to prevent the rest.