ASL isnotEnglish with finger motions; the grammar and semantics are different. Watching my assistants and staff therapists communicate with my student (I only know about a dozen signs, sadly), I've noticed several miscommunications take place because a sign can mean multiple things.
While my student can lip-read, he needs for me to enunciate slowly and clearly; often times, people at press conferences do not speak at this necessary speed/fashion.
While paralanguage is usually understood among the hard-of-hearing community, the motions used by the hearing can differ to some degree and sometimes it might appear as though they are signing accidentally something that runs counter to their intended emphasis.
There are doubtless other factors in play here, but these are the ones that occurred to me now.
Je suis méchant.