Isn't it quite possible for it to collapse? Because just looking at the area that the residential area that Solitude is built on, isn't it quite likely that one day it could collapse?

It's been like that for thousands of years possibly. Besides, it looks like it has a good foundation.

Everything collapses, eventually.
http://en.wikipedia.org/wiki/Mass_wasting

But in a fantasy setting, you have to suspend your disbelief a bit.

Earthquakes. Its not the rock face I worry about, its the bridge of the arch is my main worry. Earth quake comes, arch collapses, what is on the bridge part of the arch will go down in the water with everything on it.

Drunken Tongues/Mages, playing with Fire and Frost.

Ambitious Dragon Cultists with Pickaxes, wooden Beams, and Lamp Oil.

Anyone interested in destroying the City and not keeping Solitude as a Seaport.

Yeah I like my Proudspire manor but I couldnt live in a city built on such a disaster waiting to happen.