So, lets say I'm playing a 'pure mage' straight off the boat in Morrowind, but end up getting mixed up in House Redoran's quest line and find my character to be most aligned with their world-view. I can play through the Redoran quest line as a mage if I want, but as their society highly respects the warrior class, my character may decide that he wants to learn to wield a sword. There should be no artificial mechanism in place to stop him from putting in the effort and learning sword skills to the highest level.
On the other hand, I think there should be very strong 'soft' mechanisms in place that make it an advantage to focus on a few skills. Here are three suggestions that might work well together:
1) Make it difficult to use skills that are very low level!
A pure mage will start the game with a very low blade skill. This should have lethal consequences if he goes toe-to-toe with anything bore dangerous than mudcrabs. A glancing blow could throw the sword out of his hand, the enemy should 'read' his moves and therefore block/dodge more effectively, and hits should have a big 'random' aspect to them at low blade levels, i.e. not always swinging the sword in the direction that tha mouse is pointed.
2) Make it very advantageous to have a high level in a skill, so it makes less sense (for a given level) to have spread your skill points out over many skills.
I.e. a blade skill of 60 should be MORE THAN twice as effective as a blade skill of 30. The special moves in Oblivion were a good start here, but need expanding, and high-level spells in Oblivion just didn't feel very epic. A high-level mage should be able to do crazy stuff, and should be a priority for bad guys to knockdown/silence/drain mana otherwise they WILL be magicked to death. A high-level rogue/thief should have a deadly critical strike from stealth, and also have some way of making critical strikes in melee.
I think that in Oblivion it actually felt like a disadvantage to have specialized in something (apart from a Thieves' Guild player perhaps - but in this case you would still be eaten quite often travelling from city to city), as the game forced you into similar situations in most of the quest lines, and because magic and combat had too much synergy (no real disadvantage to spellcasting from holding a sword / heavy armour).
3) If you have a guild based around a particular skillset, for the love of all that is holy please ensure that the quests for that guild make use of this skillset. This will automatically focus the player on a few specific skills. For stuff like the Mages' Guild in Oblivion (which worked a lot like a university) have some combat quests that (due to layout, large numbers of weak enemies, whatever) are easier for a mage to survive... but why not also include some quests based around magical research (use spellmaking to achieve a particular new spell type, and demonstarte it to the satisfaction of your peers), get an object that is very difficult to access without water walking / levitation, etc. Morrowind had hard skill requirements to advance in the guilds and houses, which was better than Oblivion's completely open system, but clever quest design can do a lot better than either options.