A system where traveling to extreme freezing temperatures/high altitudes would cause slow health loss/stamina reduction allows for plausible control of where the player can go. These restrictions can then be eased as the player's skill increases (higher stamina, tolerance to cold, etc) as well add new strategy elements for low-level players such as sacrificing amour for a heavy fur coat, drinking illegal rotgut to keep warm, or choosing to play a Nord for inate cold tolerance.
The ability to go anywhere the eye can see is part of what made Oblivion great. But putting the cookie jar on a higher shelf should provide for more interesting adventure and more rewarding success.