what if your athletics and speeds were different, you wouldnt be able to keep up with them or vice-versa
I think that's more problem with the TES AI and pathfinding. This isn't nearly as visible problem in other games.
AI could "cheat" by running faster. AI could walk with you (a little behind, next to, slightly in front of you) instead of following you on rails at set range. After those mechanical, robotic problems the only problem left is the realistic one: What you do in real world if someone is too slow or fast?
I'd also like if most of the companions had their "own free will", not everyone should be like an Apprentice or slave. They could leave if they don't have any reason to be with you (of course not just run away without saying bye), they could tell you where they are going and maybe you could tell them too, they could find you, they could ask you things like "I'm going hunting, want to join?"
Yes TES shouldn't be party-based, but improving AI/NPCs is IMO easily the most important thing to improve in future. Better companions are on top of that list.