I had an idea about this awhile back, and touched on it in the Beyond Skyrim thread, but i think it's a great idea and offers some interesting opportunities. In Morrowind we had the ability to build our own stronghold, and even direct the development of Raven Rock, and in Skyrim we have Hearthfires which allows us to pick and choose building components (to a limited extent, admittedly).
I personally think that Bethesda should expand on this base, giving us a run-down town and castle to rebuild and lord over. Rather than just making it a generic 'Build this, build that, finish, move on to other things' model, they could even implement a management system where you collect taxes, pay wages, issue bounties and dispatch your ever loyal city guard to clear out bandit lairs and secure the trade routes.
There's a great deal that can be done with a 'city mini game' that wouldn't break the immersion or overall flow of a TES game.