For me, the absolute worst reaction a reader of one of my novels could have is to roll his/her eyes.
Fiction is all about creating a world, and by that I am not just referring to science fiction and alien planets. A good writer designs a little universe in which the characters live, and the goal is that the reader will for a while inhabit that universe (and if it really works, the reader continues to revisit that universe mentally even after the book is finished). This means that inside the world of the story, everything needs to be self-consistent. The plot needs to be coherent, any backstory needs to align properly, and -- most importantly -- the characters should act in a way that makes sense, given who they are. (Now, of course, there are times that a character might act illogically -- but his/her illogic should still be consistent with the character's personality. I.e., there should be an underlying logic to the illogic.)
Nothing is so jarring to me as a reader as reading along and suddenly saying, "C'mon, he wouldn't do that." I find that this tendency is especially pronounced in horror and science fiction, largely because the action in those genres often comes from people getting themselves into life-threatening binds. But even though in a horror story, you have to somehow get the main characters into the haunted house (or graveyard or castle of vampires or monster-inhabited cavern), you've got to do it in such a way that it doesn't elicit eye rolls. How many horror movies have you seen in which a couple gets lost in the woods at night, and they hear something, and one of them says, "It could be a monster. Let's search for it. We should split up." And inevitably, the monster picks off one or both of them while they're alone.
Gimme a break. If my wife and I were camping in the woods, and we heard a monster, the last thing I'd want us to do is split up. Given that I'm a great big wuss, I'd probably be clinging so tightly to her that we'd have to be surgically separated after the monster was vanquished. No way would I calmly say, "Let's split up."
I find that even my all-time favorite TV series, The X Files, was guilty of this. How many times did Mulder and Scully split up to search the warehouse for the monster? All the while using only flashlights? Several times I shouted at the TV, "Turn on the lights, for chrissake!" But they never did.
And of course, science fiction is equally bad in this regard, but usually for a different reason -- in much of science fiction, the backstory is so complicated that it's necessary that someone on the spaceship has to act like a complete moron in order for the reader (or watcher) to know what's going on. It leads to scenes like the following:
CAPTAIN: Shields up, Lieutenant! We are entering space controlled by the malevolent Bugwumps of Garbonzo-11.
ENSIGN MORON: But who are the Bugwumps, Captain?
CAPTAIN: The Bugwumps are an evil race of giant robotic cockroach entities who can control the space-time continuum with their thoughts.
ENSIGN MORON: Why are they at war with us, Captain?
CAPTAIN: Well, Ensign, ten years ago, some members of our Federation settled on a planet in a disputed region of space. And the Bugwumps destroyed it. So, now...
And so forth. Remember Wesley Crusher on Star Trek: The Next Generation? Wesley Crusher was basically Ensign Moron. And because of that, we had not only a character who was asking stupid questions, but a captain who had to tolerate his presence on the bridge, instead of saying, "Get off my bridge, you odious little twerp, I've got a job to do," which is what I'd have done, if I were Captain Picard.
The same principles are true, of course, in any genre. It's no coincidence that when someone does something surprising, people say, "Well, that was out of character." Characters in fiction have to obey a certain logic -- even if that logic is, for a time, known only to the author. It's perfectly okay if a character does something seemingly crazy -- as long as, at the end, the reader says, "Oh, okay! Now I see why he did that!" Misdirection is permitted, as is playing your hands close to your chest. But if, at the end, your reader doesn't see the pieces of the puzzle fitting together neatly, something has gone seriously wrong with your characters' motivations, personalities, and interactions. And in my opinion, there is no worse error that an author can make.