I think the main reason everyone's angry is because they think that the College of Winterhold caused a massive tectonic event which resulted in the majority of the city simply falling into the sea, wiping out the majority of Winterhold's population, power, wealth, and prestige in a single night. That's why it appears as such a small town in the game, as opposed to a big, Whiterun-sized city. Nothing will be able to give them back what they had, though the Jarl gives some quests revolving around improving the city's standing by fetching an old artifact or two.
Spoiler No one really knows why the city fell into the sea, but the College of Winterhold mages stalwartly insist that they did -not- cause it. So far the most compelling reason I've heard given, in-game, is that something theorized that perhaps Red Mountain's eruption had such far-reaching implications that it caused an earthquake that dislodged most of Winterhold all the way in Skyrim