Depends on the game, it's not like an instrument where you need about 10000 hours in order to be an expert.
I would say it took some time for me to get good at TF2 back when I started, mainly because I was new to the FPS genre, then I took a break for a while, restarted again and got pretty decent pretty quickly (in the meantime I had been playing other shooters). So yeah I think it depends, I think once you're somewhat used to the genre it becomes easier too (Age of Mythology wasn't all that hard once I started because I was used to rise of nations).
As PilaPis said Mount and Blade can be quite difficult with a lot of features, playing M&B made Age of Chivaly easier though.
But skills need to be maintained, l recently picked space marine back up and I got whooped way bad for the first little bit.
I svcked at Morrowind back in the day, I think the main reason was that I was like 11 and barely knew any english words. Didn't prevent me from trying though. DIdn't do any quests back then though.