That seems to be a real big issue and probably the main reason for most quest bugs. It seems it's not being flagged as a quest item and activating the quest on pick up. Which is what it should be doing.

If you loot an amulet that's a quest item, I expect it to activate that quest associated with it or vice versa, you talk to the person, which activates the quest, which lets you get the item.

It should work either way and in an open world I don't see why it's only the ONE way. Did they count on us to NOT go wandering of into the wild blue yonder?

Sorry, but when Giant's Toe and other Ingredients not related to anything for a quest won't exit your inventory because it's claiming it's a Quest Item, it's a bug!

Although most of what you talk about is a bug.. this is a mechanic. Granted you might not like it.. but it is a mechanic Bethesda put in to show the danger of dragons.
Only minor quest givers are killable though. All major Quest characters are or should be flagged as essential until their questline has been finished.
To be honest... one of the complaints for Oblivion was that most Questgivers were essential, in Morrowind nearly every important NPC could be killed. So me thinks that Bethesda put this back in.. because of feedback for minor quest givers. If you are annoyed by this.. Find a dragon spawning point near the town.. (wordwall, dragon symbol or dragon burial ground) and kill the dragon there. Dragons might still appear in a town, but chances are you will limit the encounters.
So annoying, depending on playstile, yes.. bug No...


eeeh... partly true... If any of the instruments are found before the Bards quest... said quest will probably bug out.. (so do not pick up named instruments)
However.... for all my characters, after finishing the quests.., they will always be stuck in my inventory.. And as I gathered this problem is among the most common ones on Skyrim.. I actually yet have encounter a single post confirming that the instruments were gone... after finishing the quests. (on lower levels the reward outweigh the inconvenience imho)
About the encumberance... This, as much as I can gather, is extremely rare... Hence why so many people claim you shouldn't be encumbered by them..
The only way I can see this happening is either the white Phial Questline or the one with the Giants toes ...(??) since these items will stack and only one will be flagged as quest item. If this is the case... have you tried eating the ingredients..?