Wait a minute, the books on shelves in the vanilla game don't go leaping off on their own. What's different about yours? Did you make your own shelves?
I'd guess that the issue is not with the books, but the shelves. If the collision for the shelves is a simple box outside the books, then they want to be back outside it and will do that. The collision surface needs to be more complex, to allow the books to sit on it.
No, not at all. They're vanilla shelves, but I've put all the books standing upright. When they're Havok'd they tumble over each other, and then fall out onto the floor. I might try placing them closer together, but then they could still
try to tumble over each other and end up exploding out onto the floor.
EDIT: Yes!!! The reason that they were falling out was because their collision boxes were knocking into each other, forcing the books apart. Forced into the sides of the bookshelf, they had nowhere else to go but out onto the floor!!! Now that I've moved them further apart, everything works fine!
However, it is extremely annoying that I spent about three hours of hardcoe thinking about this when the solution was so simple!