Part of it comes down to actual prejudice, part of it people being a-holes (prejudiced or not) and part of it is people being oversensitive.
I agree with you that the over-sensitive are a burden, particularly if society tries to compensate by making efforts to not annoy them. In the story where I accuse myself, notice I have no evidence at all that she was annoyed by my behavior. If she noticed it she didn't let me know. Her reaction could have been nothing, she got annoyed and got over it quickly, she got annoyed and stewed about it for days, or she put me on her list of men who will be lined up against a wall on the day the wymenists take over. Well, probably not that.