The difference is:
Oblivion is a open-world Roleplaying Game, Freelancer is a Combat Simulation Game. Thus, they had far less to worry about! In Skyrim, a too sophisticated AI could actually break the main quest or other quests, by doing things that was unpredicted of them. An open-world RPG have to have predictable AI to make it work, unfortunately.
I'm going to answer this with another quote. Sorry if that's against forum rules/etiquette.
One would hope they could have checks and balances. If things go too far into Sheogorath's madhouse, the forces of Jyggalag step in.
For example, if a towns person begins attacking others, if it is a city with Guards, the guards automatically are as reactive as they are to the player, stepping in, but not attacking the NPC, only saying the typical, "stop right there lawbreaker," or if its a drunk or just a brawl, 'all right. Lets cool down citizen, or else you'll be sleeping it off in the prison." The game could be programmed to have the citizen comply at least 95% of the time, ensuring that the towns never descend into total chaos.
Controlled Chaos, I believe is the usual term. Failsafes, gauranteed pre-programmed stabalizing responses that fit seemlessly in with the gameplay, and ensure that dynamic NPCs don't become game breakers.
I find my only problem when attempting to completely complete Oblivion is that I have a life outside the game or I get bored because the task is monotonous. The random attack by dragons in Skyrim WILL cause a lot of quest related NPCs to die, unless Bethesda makes NPCs teleport away to safety while the player deals with the problem. Radiant Story looks like it will prevent these issues from being a problem, so I don't see why we can't have improved AI to boot.
To be honest, I wouldn't mind if some quests (with exception of the main quest) were unsolvable. If it's a game mechanic, it's all well and good. If it's because of a glitch, that's not. But in a game like Oblivion, like Skyrim, the player character is just another NPC trying to carve his name into the pages of the elder scrolls, and the game designers can't possibly predict everything the player character can and will do.
P.S. how does multiquote work? I can't sort it out without opening two windows.