Exactly.
Perhaps next Elder Scrolls game, (assuming that it will be on another generation of consoles) will have new physics engine that makes this possible even with a game as detailed as the elder scrolls
It could have been done now anyway. There's a game where physics is incredible, the graphic and detail of everything makes Skyrim still look bad, and not only that, you can go anywhere you want within the world, just like Oblivion and Skyrim. In fact the area size, in perspective, is probably about twice the size of any TES game other than Daggerfall', "Just Cause 2". The story is crap, voice acting and dialogue is horrendous, but the graphics, gameplay and physics are awesome and the world is huge. After playing that game, I always wondered why Dragon Age and now Skyrim couldn't look like that in detail.