Athletics doesn't exist anymore, from what I've read, so that shouldn't be much of an issue. Never-the-less, in Oblivion, we had seven attributes linked to skills (As luck didn't link with any skills), which meant you had three skills per attribute, which meant three-hundred opportunities to assist in levelling that attribute. Even assuming you managed to 100% completely max out your athletics, that still left you with two-hundred chances. I just hate the fact every time I read a god damned interview, some feature has been cut, and I can't understand why, until I remember how stupid the average person is...
...And then remember that half the population is stupider than that.
On of the problem in Oblivion attribute system was that, in Oblivion when you leveled your Blade skill to 100, you won at most 50 points of Str as a result. So, once you were capped in Blade, the only way for you to improve your fighting capabilities with blades was to ... to train with Blunt or Hand to hand which is exactly the thing you are against.
Besides, a basic and plain attribute/skill system like in Oblivion was NO WHERE NEAR the complexity of a good branching tree perks system if done well.