First of all, I think you are referring to Legacy Lost (the quest after Separated at Birth).
Occasionally, the three ogres wander off and when you arrive at Weatherleah there are no ogres to be found anywhere - even though you are told that they need to be killed. Check the surrounding area. They've been known to wander long distances to attack passing monsters and hunters - even as far away as Hackdirt and beyond. If you continue to have problems with this portion of the quest and you are on a PC, you can use the console command
setstage MS16A 40
to advance the plot to the next stage. If you are not on a PC it is advisable to load an earlier save until the ogres are there. Do not continue with other quests before completing this one because you will not be able to obtain the Escutcheon of Chorrol, the Honorblade of Chorrol, or begin the quest Sins of the Father.