Stats and certain skillpoints, though game mechanics, are representative of inheritable traits. Orcs are naturally stronger than the other races, Nords are naturally tougher. I also believe that even without training, Redguards have a natural talent with arms. And no race will ever jump higher than a Khajiit, or swim faster than an Argonian. Elves have a greater affinity for magic than non-mer.
And with the example I quoted, again, I say he got it backward. What he listed as traits are the racially-powered ones, while saying he wouldn't recieve minor physical traits. I said he wouldn't get those race-based traits, and would get the father's physical ones, like dusky skin and red-tinted eyes.
Stronger, tougher, talent and aptitude, are very vague terms. You can not say that every Nord will be tougher then an Imperial, because there might very well be an Nibenian mountaineer who is going to outlast a Nordic trader from Solitude. Because of this, to measure these racial traits, you have to you average over all individuals of a race too see the difference. Yet doing this, also means you include their cultural preferences and geographic influences.
So at this point you have to wonder, if it isn't because Magic plays a much greater role in the live and upbringing of an Altmer that they have a greater affinity and weakness too it. Just as collectively living in the mountains is going to show a slant everything towards a more sturdier constitution.
That isn't to say that there are no physical difference at all, but rather that their influence towards their racial stereotype among the non-Betmer races is over stated.
Now I don't think dusky skin and red eyes are subtle or minor traits. When I'm reading that I'm thinking about the slant of the ears, minor changes to the bone structure. Basically so little of everything that you it would be easy to confuse a half-breed with the original. After all the book is also the defacto explanation for a lack of half-races.