The speculation would be great. Especially since the non-readers are already speculating that Jon Snow is Rhaegar & Lyanna's child and not Ned's kid.
How could they? From that one conversation Robert and Ned once had in S1 about it? I am sorry, I don't buy it. Maybe it's my being no big fan of this theory in the books to begin with talking, but I don't see how anyone could come to this conclusion by ONLY watching the show. Maybe by studying every episode in a dozen viewings, but not the regular viewer who sees every episode once or twice.
You mustn't be afraid to dream a little bigger, darling.