If that were true then the Bretons would look exactly like one of their two ancient parent species, but they don't fully resemble any other race of men, nor do they resemble elves. A hybrid offspring may more strongly resemble the mother but there is definitely a physical attribute or two from the father as well, and over many generations of inter breeding you get a unique appearance.
I also believe there's no reason to assume that an offspring between an Orc and a Nord would be sterile, for the same reason - descendants of Elves and Men have already been determined to produce entire races of fertile offspring.
I said that the traits of the father are passed, just subtly. Given enough time and a significant population of two races living together with no gene immigration and emigration, and complete random breeding, the two races will eventually become 1 race. This is exactly what happened with the bretons, a fact that seems to be conveniently forgotten a lot. And that took many hundres and thousand years to accomplish, so it's definitely not an every day occurrence, especially now that the above conditions are pretty much gone in modern Tamriel.
And never did I say the resulting half-breed becomes sterile. If a nord and an orc produced an offspring, it would indeed become fertile still. And that is mentioned in the book I linked, so that last statement was not really needed. What I did say is that vampires are supposed to be sterile, so the case of the Grey Prince is a giant glaring "WTF JUST HAPPENED!?" Even the father vampire was REALLY surprised that his coupling with his orc maid actually got her pregnant.