Caring about or showing an interest in something is usually fine...showing an excessive interest in it is usually considered "weird," unless you're among a group of people who all share an interest in said thing. And "excessive" usually means "anything that might take effort to learn." So with poems, showing familarity and appreciation for them seems to be fine, but being able to recite them implies you're some poem-loving weirdo who sits there at night with volumes of poetry, committing them all to memory.

Yes, this is silly. But that's how it comes across to people.

Or at least, that is my understanding of things (I'm not the nest at picking up social cues either). I once got mocked as "Miss Walking Library" for the offence of knowing the constellations and the names of various major stars in the night sky (and I wasn't showing off, I was just trying to help someone who was actully trying to locate a certain constellation). Apparently such knowledge is "incredibly weird."