I'm thinking that the Akaviri had roots in Atmora due to their apparent obsession with Dragons, their knowledge of the typically ancient Nordic legend of the Dragonborn prior to Pale Pass (and maybe as far back as when they were still on the Akaviri continent) and their apparent obsession with trying to invade through Skyrim.
They even treated the citizens of Skyrim with the utmost respect during their occupation of Skyrim before Reman defeated them at Pale Pass. Maybe they had the same kind of ties to Skyrim that the Atmorans did (being conceived at the throat of the world?).

Is it possible that they were more like a splinter group of Atmorans who didn't want to worship the Dragons and evolved separately on Akaviri?

We never did learn where the men of Akavir came from.

Seems reasonable that another group of Atmorans set sail and wound up a bit too far to the east.