I can't find anything that confirms it one way or the other, but I have a feeling that in order to get the relevant conversation topics ("Disappearance of the Dwaves") you may need to have started the quest.
If you have the books, you could try going straight to Yagrum with them. It's probably the best chance of having this work.
On that last note, UESP does say this: "Of course you can take a shortcut and speak to Yagrum Bagarn straight away once you have The Egg of Time and Divine Metaphysics...." But again, i don't know if that means it's a shortcut through the quest (ie, you have to have started it but you can cut straight to him) or if you can just take him the books without starting the quest.