You may use both shelves. Ensure you're pointing at the right shelf when activating.
the activation zones seem to overlap somewhat, so even if you aiming at the lower shelf it sometimes activates the upper one. Try slightly different spot on the shelf...
Hmmm, I could almost swear that I've meticulously scanned every inch of both bookshelves in my Whiterun house without any luck, but I'll go back and try again to see if I can get both shelves to work instead of just one. Then again, my weapons rack in that house is bugged for sure, which seems to be fairly common glitch on the consoles (I play on the PS3), so I may just be out of luck.