What is annoying about jumping high and running fast?
Actually, I found the leveling of athletics could be annoying. In vanilla Oblivion, you had to watch and control what skills ranked up and when in order to ensure you were getting a balanced level with all the +5s you needed without wasting future attribute points. It's basically too annoying to start walking everywhere, so you are going to always be gaining athletics, and you gain it pretty fast early on in the game, when you are probably wanting to focus more on skills like block for their endurance (and permanent HP boost).
Likewise, I would have to avoid jumping, because that way, I could "save" my acrobatics levels for when I needed the attribute boost.
Have I mentioned yet how much of a relief it is for the vanilla Oblivion leveling system to be gone? That thing forced me to break off from adventures to sit in the basemant of the mage's guild repeatedly spamming low-level mysticism spells or boxing with my summon skeletons to gain the stats I needed, lest I face the wrath of sub-optimal leveling.
While I'm on that topic, keep in mind that people are already complaining about how some skills are not useful in combat, but level you up (and hence, level up enemies), anyway. Ideally, for this game to be balanced, roughly equivalent amounts of ranks in various skills should contribute roughly equally to your success in completing objectives. That means that being able to summon monsters, swing a sword, block with a shield more effectively, or craft better armor should all contribute to a viable playstyle.
Pure rogues are already hit with several skills that just don't have the ability to bypass many fights. Stealthing past a threat is great and all, but for speech, lock picking, and pick pocketing to all be separate skills leveled on their own, they should be justified with serious advantages in combat or ways in which you can bypass needing those combat skills in the first place. That's not impossible, it just takes quest design like in Arcanum or Deus Ex. (Of course, absurdly high pickpocketing may actually be pretty abusable if you can steal a bandit's sword before combat begins, but speech really needs more serious uses...)
So then, if athletics were involved again, would skill ranks in it really be worth as much as more sneak and one-hand for your backstabbiness?
I'd rather see enchanting equipment used or perks or potions or the like that just give you speed or jump ups. A "jump" alteration spell would actually be pretty cool, as well.