The Tsaesci are described in two common ways:
- Humanoid, but covered in beautiful golden scales,
- Humanoid upper body, but serpentine lower half, covered in golden scales.
The common being the golden scales, which could more likely be a refrence to the scale like Akaviri armor. Their name also implies snakes and serpents, which would fit with the reference to several artifacts like the Ring of the Vipereye. Also, Septim V's invasion of Akavir was demolished almost instantaneously by Tsaesci. These alone make the Tsaesci the likely invaders.
The Tang 'Mo are monkey like, and are only known to fight with the Kamal that invade every once and a while. The Kamal are demons apparently, and thaw out of their frozen hell every once and a while to attack the Tang 'Mo, and dont seem to be the type to fight with the Tsaesci. The Ka Po' Tun likely, due to their love of Dragons, dislikes the Tsaesci, who apparently killed all of the black dragons and most of the red dragons, and would likely not fight with them.
Leaving two species from Akavir to invade Tamriel:
- Mankind, in some form
- The Tsaesci Empire
The major thing being, in the http://www.uesp.net/wiki/Oblivion:Akaviri_Diary_Translation, which suggests that the invaders where bi-pedal, due to the messenger's leg being injured, so you can assume either:
A) Tsaesci are not snake-like in body. B ) The term that they "ate" man, more likely means that they merged societies with the men of Akavir, and Man fought in their armies, or where sub servant to them.
I like the second choice because apparently the Men of Akavir where quite strong, and would likely not go down easily. If you consider that the Tsaesci are apparently Vampiric in nature, they could have some kind of vampire infection to mankind, or adapted man into their society for their skills at war.
The invasion of Akavir was stopped at Pale Pass, at the border of Cyrodill and Skyrim, implying that the Tsaesci fought in Skyrim.
So in the end, the people who invaded most likely had something to do with either the Tsaesci or men controlled or allied with Tsaesci.