Why is that? Why would the population become progressively less elven over time? You're assuming that they primarily breed with races of men. If they breed equally with men and mer, they wouldn't become any less elven over time. Or if they breed primarily within their own race, they also wouldn't become progressively less elven, unless more elven traits are sixually selected against within the population.
May I refer you to: http://www.uesp.net/wiki/Skyrim:Racial_Phylogeny.
Admittedly, much of my stance comes from the fact that they're commonly referred to and appear as a human race. This indicates to me that the interbreeding was primarily elven men with human women--in the Elder Scrolls universe, their offspring would still effectively be human with a touch of elven leanings. I'd argue that applying genetics as we know them to this particular IP would be a mistake.
That's the issue here: there are no true "hybrids" in Nirn. There are humans with a hint of mer and elves with a hint of man, but no true half-and-half...er, people. So from the very beginning, Bretons wouldn't have had an equal man/mer split.
(I apologize for how disjointed my responses are--it's been a long day.)