Perhaps none of them wanted the city bad enough. I don't think it has that much influence in the world. The only thing that makes it special is the guardian. It's destruction would remove the citys only claim to fame.

Rand actually has a real reason to take the effort to destroy the guardian. The borderlander leaders are in the city and he cannot meet with them without significant risk to his life (and by extension the world) unless he disables the thing.

The false dragons were probably more interested in conquering major cities like Caemlyn or Tear. But you are right, they COULD have taken FM if they had really wanted to.