Hmm - I thought my Morrowind Dark Elf was average height and the Dwemer spirits always seemed much shorter than me ... no idea about Yagrum Bagrum as he was using his spider-legs.
It was hard to tell with Vivec and Almalexia as they were always using Lev.
Are you sure Almalexia levitated?
In any event you have the option to fight both of them and during those foot battles they're at least as tall (if not slightly taller in Almalexia's case) than most Dunmer NPCs.
And the Dwemer spectres are just that, ghosts. If you notice, the other ghosts in the game are not exactly human sized either. Not a good indication of their size as corporeal beings, but judging by weapons and armor that are easily equipped by all other races I'd say they were of average height.