Nancy Pearl - reader's advisor extraordinaire, author of Book Lust, and model for the librarian action figure - came up with the Rule of 50: "If you still don't like a book after slogging through the first 50 pages, set it aside. If you're more than 50 years old, subtract your age from 100 and only grant it that many pages."
More and more, I'm discovering that life's too short to read novels that I'm not enjoying, or simply am not in the mood to read, even if I feel I should read them. Until recently, I kept bookmarks to mark my place in novels I never got around to finishing. For example, my copy of Dorothy Dunnett's King Hereafter still boasts one of these, ten years after my valiant attempt at reading it (please don't send me hate mail because of this). I got about halfway through. I don't know if I do this to assuage any possible feeling of guilt on my part, or if it's my lame attempt to avoid the obvious truth - that there are so many other books out there that, most likely, I'll never get around to picking these novels up again.
I don't have a hard and fast rule for these things. Review books, I'll slog through regardless of enjoyment (but fortunately, I do enjoy most). And I'm occasionally rewarded for my efforts - sometimes novels surprise me and turn out to be better than originally anticipated.
My latest two rejectees are Catherine Cookson's The Silent Lady and Hilda Lewis's I, Jacqueline. The former was Cookson's final novel, which she dictated to her husband from her sickbed - she was very ill at the time. I found the plot convoluted and the characters unappealing, and I really wonder if it would have been published if Cookson wasn't the author. I made it through about 30 pages. I, Jacqueline, a biographical novel of Jacqueline, Countess of Hainault, started off well, but the heroine was of the TSTL type - so naive and wishy-washy that I spent most of the novel (the half I read, anyway) cringing from her poor decisions.
It's possible I just wasn't in the right mood to read any of these novels. But I'll probably never know.