Yep, unless you're on a PC and someone releases an anti-quest-item mod. Oblivion and FO3 had the same issue- you're stuck with your quest items. It's especially annoying when the item has weight.
There's also a PC mod that lets you click on the display stand and add them directly. You get no caps, but they don't clog up your inventory any longer either.