First house (yay), first bookshelf (hooray), first problem.

Bigger books (like the Wolf Queen or Wabbajack, ones with green and gold bindings) work just fine, they stand upright and don't tip.
Smaller books immediately fall over, and don't stand straight even if I fill the bookshelf completely. I can somewhat prop them up with the larger books, but it still looks a bit dumb.

Any trick to filling the bookshelves nicely?

My brother actually had a great idea for this when the SDK comes out: Bookends.

Just heavy, decorative items you could find/buy that are heavy and labeled as "books" so they can be placed on the shelves to keep small books from flopping over.

Without mods, and without doing what you're already doing the only thing you can really do is completely fill the book shelf... Or manually place them.

Have you tried filling the shelf with only large books or only small books? You may not like that aesthetic either; it may look kind of monotonous.

Only large books are perfect. Only small books topple. And combos of large and small books topple too. Curse you, gravity.