From a letter related to the Daggerfall quest Mantella Revealed:
"Numidium was Tiber Septim's secret weapon in his bid for supreme power: a thousand foot tall automaton, a golem or an atronach of sorts powered by a gem called the Mantella. The Mantella was infused with the life force of TIber Septim's Imperial Battlemage, and with it, Septim crushed all who stood in his way. After the complete and total defeat of all his opponents, Septim began using Numidium to crush the neutral royal families of Tamriel so that he could enthrone only persons he knew to be loyal. His Imperial Battlemage was furious at this use of his creation, and fought to reclaim the Mantella. In the ensuing battle, both the created and the creator were vanquished: the heart they shared blown out of this reality into the netherworld they call Aetherius. Numidium's body was scattered throughout Tamriel..."
It's not clear how devastated Summerset was by the use of Numidium, nor is it clear whether the Altmer knew the Numidium's body had been scattered. They may have thought that any rebellion they could muster would be swiftly crushed by it.
Anyway, it can be presumed that Tiber made it a priority to enthrone loyal persons in Summerset promptly. And about a century later, the Empire helped the Altmer repulse a Maormer invasion, which probably helped foster good relations.