Yeah, but it's not the numbers that are making it an RPG, it's the fact that you can develop a character and influence the change of said numbers, but traditional RP'ers have gotten so used to the idea of the numbers being what makes their character that they can't see it being an RPG without the numbers being visible.
It is about the numbers. The numbers are vital. But you're right that these numbers don't need to be visible to the player at all.
Say for the next TES game Bethesda went a step further and made it so your character does clearly develop, they get better at things, gain special skills, basically you can do anything you can in Skyrim or Oblivion as far as character development is concerned, BUT, they made the entire numbers system that governs how your skills progress, how well you fare in combat etc. invisible to the player.
There's a difference between hiding the numbers from the player, and getting rid of the numbers completely. Take Athletics for example. Bethesda haven't just hidden that skill from the player - they've removed it from the game. It's not possible for me to make a character who's naturally faster than other characters.
Of course there would be complaints that 'I have no stats, this isn't an RPG!!', but your character is still developing as they would with the stats. It's just not visible to the player so it is free to develop more naturally rather than this idea that there must be some arbitrary numbers to appease those who can't see past the underlying numbers game of RPGs.
If the numbers are just hidden, but not removed then the game is still an RPG. I can understand why people would still complain though. The thing you have to appreciate is a lot of people actually like playing around with stats and carefully planning what kind of character they want to make. The character creation aspect of the game is really enjoyable to them. Sure, if they just play the game and let the character evolve naturally they may get the same end result... but they're missing out the fun of actually creating a character build beforehand and then thinking about it carefully throughout the game as they progress.
To be honest I think this is the next step for RPGs, a game with all the character development of a traditional RPG, but rather than having to have pages of stats and skillsheets you just see that "I trained with swords for x hours, look how much better my character can use them, he's clearly a much more proficient warrior" rather than, "I trained with swords for x hours, my sword skill has gone up, that makes me better".
I would agree that it's the next step for RPGs looking to get closer and closer towards virtual reality (which includes TES). In a truly immersive RPG, there should be no UI at all - which includes stat sheets. But not all RPGs need to go in that direction. I think there's room for immersive action RPGs like that, as well as more traditional RPGs which keep the stat sheets, tactical combat, etc.