You guys haven't done much game development, have you? Let me fill you in.
First off, "Gamebryo" is not a "game engine", it's just a piece of middle-ware which makes up a game engine. For example, Civilization IV uses Gamebryo and looks NOTHING like Oblivion, which uses a similar version of Gamebryo. Nor can an Oblivion mod be ported to Civilization IV, even the modding utilities are incompatible.
Gamebryo gives you a scripting engine, XML parser (Bethesda uses the XML very lightly, mostly only for the UI) and an extensible 3D renderer. The rest, Bethesda has to source elsewhere or make itself. Civilization IV has no spells, but Oblivion has no turn based combat, Bethesda (and Firaxis) had to make that itself. Same with Civ's netplay (though Gamebryo supports that natively) and so on.
To actually make Gamebryo into a real game needs a ton of work because it's not a game engine, it's part of one. It's much more accurate to say "The Oblivion Engine" than it is to say "Oblivion's Gamebryo Engine". Oblivion adds Havok, Facegen, Speedtree and other "engines" even before it gets anywhere near the high-level stuff that actually makes up a game design.
Most engines these days have a scripting interface, it makes development much easier. On with a few salient points:
It's not a matter of upgrading - they'd have to rewrite massive portions of their engine to support the new features in the latest Gamebryo - and they've likely customised, or added their own features to the version they have, making it quite impractical.
Between Morrowind and Oblivion, Bethesda did just that.
You know, "huge draw distances and sprawling vistas" does not equal "open-world". It merely presents the illusion of it. Would you call Borderlands an open-world game?
This is exactly how Oblivion and Fallout3/NV works. That distant land? Yeah, it's low-detail, low-poly fakery to present the illusion that the cells are loaded when they're not. The only part of the game that's loaded is the five cells immediately around the player. Even the AI doesn't run until the player arrives, then as the cell loads, the AI quickly catches up - Unless the player is in the same cell or the cells are "local", then NPCs do not exist short of a very basic script which lets quest targets move around (and that doesn't always fire properly). This is why NPCs would never die when following you if you fast travelled, even if a whole den of bandits was between you and the target.
The whole "stumbling upon a battle"? Yeah, you did that so much because the battle couldn't start until you loaded it.
I'm pretty sure if idTech 5 was able to handle the types of open worlds that Bethesda creates, they would've already jumped on it and announced it to the world several times that they would be using it. Hell, at least I'd boast about using idTech 5.
Of course it can. Probably better than Gamebryo can. Handling cell and area loads is for the developer, not the game engine. The game engine just has to draw it. Way back when Morrowind was the new and hot stuff, NetImmerse (renamed to Gamebryo later) was the only decent middleware which could handle the kind of draw distances needed and it did that extremely badly, needing the distance clamped and hidden behind fog. Now every engine and its dog can draw more or less to infinity.