"Old Elf" could be phonetical for Aldmer, since "mer" (which is pronounced "mare"). Ends up the same, though, doesn't it?
It could also be a result of Aldmeris being the root language of "Tamriellic Common", Much like Latin is the root language of Italian, French, Spanish, and English.
This would result in several cognate words, much like happens with our world, but also with different grammar and syntax. The ashlander in question could well be speaking real Aldmeris, while the player is speaking Tamriellic Common. It would sound a bit jaded, with perhaps improper grammar, as it is a related language but not the same.
EG-- Italian and Spanish have some very similar words, and can sound almost the same to the untrained. EG-- 'Gratsi' and "Gracias" (Or german and English, "Danka" and "Thank-ya")