1) the book doesn't work as a whole --
many amazing writers fall into this category, especially if you're in a critique group. you write a killer 10 pages every two weeks, but those pages don't work when you put them together. you are probably also writing whenever you can grab a few minutes, a method that can create a disjointed book of forgotten threads and many moods.
2) you've written the book of your heart, and nobody wants it --
unfortunately the book you really want to write might be the hardest to sell.
3) lack of mass market potential --
this is related to number 2. your writing is wonderful, editors might love your submission, but it doesn't have broad market appeal.