Is there some point in the quests where the quest is finished and the quest items while still in your possesion become non-quest items and can be put down? I haven't done that quest yet but I would like to find out.

Black Books remain in a characters inventory forever.

The Dragon Elder Scroll can be sold to Urag in the College of Winterhold Arcanium. The two Dawnguard Elder Scrolls can be sold to Dexion if a character sided with the Dawnguard.

If you're worried about the weight, don't sweat it. Quest items don't count (even if the quest is done). All those black books and elder scrolls won't weigh you down.