This is mostly fun and speculation here. It would be cool if this issue got finally ironed out for good.

A lot of the lore regarding the exact origins is rather vague and a lot of room seems to be open to criticism(perhaps deliberately so).

That mixing between nedic and mer happened seems to be something of a matter or fact, yet, the extent to which it even happened is extremely vague and it is entirely possible that actual Bretons with mer ancestry could be a small minority of those descending from very ancient lineages.

I've seen it suggested that the particular physiology and personality quirks of the Breton's are an indication of them possessing Mer ancestry, but is this really the most likely explanation?

At the moment they seem to be shorter and lighter, intelligent and have a high natural resistance to magic. Higher than any mer, including the Orcs.

The Aldmer were supposed to be tall, but the Bretons are pretty short. Both have a light build. The Aldmer, who are supposed to be like the Altmer, are intelligent, like the Bretons, but intelligence varies quite considerably among the races. The Altmer have quite consistently shown that they actually have a weakness to magic despite having greater magicka reserves than the Bretons yet the Bretons are very good at resisting magicka attacks naturally.

I don't think it fully adds up that the unique qualities of the Breton's are a result of very distant admixture with Aldmer or Altmer. If anything they look like a population of Nords who have had their most aggressive and foolhardy members with the least natural tolerance to magicka attacks being killed off by their Mer masters, with the less physically imposing, more cunning, cautious and magically resistant members living on to propagate their distinct line. Build upon this a history of these people being surrounded by Mer culture and you could eventually have something like the Bretons emerge from this with or without any real influence from their Elven ancestry.

To put it another way, the Bretons could, if anything, be what happens when you put Nedic people in a magical pressure cooker where they either get tough or die.

Is it more a case of hybrid origins or mere natural selection that brought the Bretons into existence?

I personally think the natural selection angle is plausible and actually quite interesting, as it could also explain the evolutionary development of the other races which cannot always easily be attributed to aedric/daedric intervention or race mixing.