It was seemingly implied that the silent aliens were the leaders of the whole Silence group, or at least the bosses of the part of it we've seen, and the bosses of Kovarian. In that case, they're the ones who believe that the question must not be answered, etc. Kovarian referred to their conflict with the Doctor as an endless, bitter war, but I wonder what made it bitter. Perhaps the fact that the Doctor essentially programmed the human race to kill the silent aliens on sight, essentially causing the deaths of untold numbers of them. But clearly the silent aliens were already working on making Melody Pond into a weapon to kill him, even before he did that.