I went to the bard's college and picked up 3 quests to retrieve specific instruments for the bards. I located and returned the items, but they are still in my inventory. This also applies to Gallus' Coded Journal, 2 amulets of Talos and the Gauldur Amulet Fragment. These items have been stuck in my inventory and I cannot get rid of them, even though the quests for the have been long since completed. Bethesda, you guys need to get on the ball here.

Known bugs.
Check this thread: http://www.gamesas.com/index.php?/topic/1298524-xbox-360-bug-report-topic/
In particular, see the opening post and scan down to Items.

I have the same problem with the same items with the exception of the Gauldur Amulet Fragment, there are three pieces to it, the item you get out of them is amazing. When you picked up the Fragment there should have been a note on the body as well which gives you a quest to do which leads to the other fragments.

I haven't come across any of these problems. Also what Leg10n666 said. The Gauldur Amulet Fragment is apart of a quest that you just haven't completed yet.

Did you do the Bards quests.. I'm pretty amazed if someone didn't encounter this bug..

Yup. All my Bard quest items are stuck in my inventory.

I have the music items bug also, I've also got a few book items stuck in the inventory related to finished quests, all they do is clutter the place up, no weight penalty is attributed to them.

I also have miscellaneous quest entries in my quest folder for compleated quests, I had one quest popup stating I helped 6/5 people in Solitude because it's still showing 5/5 in the misc. section lol.

I haven't done the bards quest line yet. I started then went and did the thief line quests and then went to do other quests. I'm all over the map on quest lines.

I posted something about this over in the PC thread about console commands fixing broken quest items. Here's my idea. Not trying to spam.

Beth needs to implement some feature like this for consoles. Like holding down X (Drop button on Xbox) button for 5 seconds will force drop a quest item and ask you if you really want to drop it.

Have fun with the undroppable instruments