1. Make levitation hard to purchase or learn.
This could be done by; A. making it so that you need to meet a certain rank in the Synod or College of Whispers (or whatever mage faction(s) they have in Skyrim) before a trainer will even offer to teach the spell, B. making the spell very expensive, and C. making it so that only a handful of trainers in the whole province will know the spell and be able to teach it.
2. Make levitation have a constant drain on your magicka.
Levitation would be severely limited if there were, for example, a 5 point/ second magicka drain. That means that if you have a magicka pool of 50, then you would be able to levitate for 10 seconds. At 100 points, you could levitate for 20 seconds. If you become a god of a mage, and have a full 500 points of magicka, then you could levitate for 100 seconds (a total of 1 minute, 40 seconds). That isn't even taking into account that you may want to cast a slowfall spell, or toss a few offensive or defensive spells while levitating.
3. Bring spell failure back.
Assuming you are able to find the spell, and you have a high enough ranking among your guild to learn the spell, and you have enough money to buy the spell, and enough magicka to cast it for more than a few seconds, another limitation could be that the spell simply fails because you haven't quite mastered how to use it yet. This would add a tactical element to the game because now it becomes a decision between (probably) nuking your opponent with spells you have a better mastery of, or (maybe) trying to gain the high ground against your opponent by levitating to the next ledge, or (likely to fail) 'flying' away altogether. It could go a step further and be so that spell failure caused damage to your magicka pool, fatigue, and/or health.
4. Make magicka regenerate more slowly over time (or not at all).
If it took more time to regenerate magicka (e.g. one or two in-game hours to fully replenish), or if magicka didn't regenerate without resting, then I think people wouldn't be so quick to use levitation for every mundane task, particularly if some of the aforementioned points were implemented. In fact, I believe that people would be a little more conservative with their spellcasting altogether if magicka didn't flow so abundantly as it did in Oblivion.
The point of saying all of that is to articulate the point that levitation can be implemented in a way that is difficult to abuse if Bethesda really wanted to.