I don't think the walled cities in OB were a result of the engine not being able to handle them. I think the biggest reasons was that they had trouble with npc's (probably the guards more than any else) constantly running into the wild to chase or get away from beasts and monsters. as well as keeping things like Ogres and minotars from killing all the non essential characters. then after all that there was the problem with to many static objects that would have severely affected the speed of the game. Not in every city, but Skingrad was the worst and even with the cities being in a excluded cell my frame rate dropped now and again.
Vivec did not have any walls and it was super massive. not on gamebryo I know but its still worth pointing out.
While RAI probably did have an affect on the control of things, the NPCs themselves played a more prominent role in closed cities as the time it took to render them. Any decent amount of actors on screen (or in the player's rendering line of sight, which doesn't ignore npcs behind walls, etc) could cripple Oblivion. And the same goes true for the architecture.
RAI has hopefully gone through at least some basic refinement, so I'm hoping it won't be an obstacle to open cities in Skyrim.
Vivec was super massive, but only in the size of the cantons, which were themselves pretty simplistic. All the real render-intensive objects and actors were in interior cells, for the most part. And of course, Morrowind's render distance was quite a bit smaller than Oblivion's. Less data to load for the immediacy, and it didn't have to worry about LOD issues.