After comparing my High Elf's height with fellow Altmer npcs by standing next to them, I noticed that he's the shorter one in most cases. Some actually tower over him. He's still taller than npcs from other races but appears to be a shorty amongst his kind. Have you noticed this when playing as an Altmer?
My Altmer is about the same height as Taerie, but Ancano and Ondolamar are taller than she is. I wouldn't mind her being shorter, she hates being taller than her spouse!