The people who live in the house and can repair it (mortals), for the house was built for them and has no purpose in their absence for either the builder or the destructor.
The builders are able to build and rebuild - such is their purpose. The destroyers are to destroy and ruin the work of the builders, but once their deed is done, what is left to destroy?
Awesome proverbs aside, I believe we haven't sufficient information about the Aedra (thanks for downplaying ALL of them except Akatosh/Talos, Oblivion) to form a conclusion on this.