I get the feeling that teleportation wasn't common in the Third Era; mostly used by mages and adventurers. It could be most citizens didn't have much contact with magic unless they had business with the Mage Guild or with priests.
The meteorite that made Arizona's Barringer crater was only about 54 yards across, according to the link below, and it left a crater a mile across and ten stories deep. It's said that a meteor hitting water is worse, because of the tsunamis that would occur. Probably the Ministry was bigger than what we saw in-game anyway.
http://en.wikipedia.org/wiki/Barringer_Crater
I'm reminded of the slaves' chorus from Nabucco, "Va Pensiero", though that was for the Babylonian Captivity instead of the destruction of Jerusalem. http://www.youtube.com/watch?v=4BZSqtqr8Qk
http://en.wikipedia.org/wiki/Va_pensiero