The Ka Po' Tun need to kill or commandeer the snake peoples naval fleet in order to invade. All there assets have been compromised to deal with the snakes after their alliance with the Tang Mo. Just a guess.
Well, apparently it's simpler than that: the Ka Po' Tun are just bad at seafaring (they're cat people after all), so they never set foot on Tamriel.
Yes, they also have an alliance with the Tang Mo, the monkey people, but that race is fairly primitive and involved with their own local struggles against the Kamal. The Kamal are a demon race who dwell underground most of the year and emerge every year and try to conquer the Tang Mo, so far they've not succeeded. They tried to invade Morrowind once, but were defeated by Amalexia and the Underking.
The Tsaesci or "Snake Palace" hunted the dragons to extinction in Akavir (think: Dragonscale Armor). The black dragons allied with the Ka Po' Tun before supposedly going into hiding to avoid the same fate as their red cousins. Dragons of Vvardenfell were driven to extinction by invasive cliffracer populations. Any remaining dragons are rumored to live high on isolated mountain tops (Sounds like Skyrim to me, just sayin')
The Tsaesci are the only Akaviri race to make any decent inroads on the continent of Tamriel. They rely on avaviri goblins for their food and labor.. In 1E 2703 they tried to invade Tamriel by way of Skyrim, but were defeated by the emperor Reman I at Pale Pass. He allowed the defeated Tsaesci to serve in his war against the elves and they eventually integrated into society (think:
Imperial Dragonscale Armor), and even took the throne for 400 years until the usurper's descendants were eliminated by the Morag Tong in 2E 431.
Here's my current prediction: The Tsaeaci see the empire weakened after the oblivion crisis and try to invade again through a similar route as before, the end of the Septim bloodline leaves the provinces fractured and a serious power vacuum (It's as good a time as any to invade). a possibility is that the dragons come out of hiding, maybe to aid in a resistance or possibly be involved in naming the Dragon Born as a hero who will fight for the freedom of Tamirel.
response to posts that came in while I was typing: I would love to see a return to Morrowind spirited creature design. The Underfyrke, horkers, those were gold. Minotaurs? not so much. It would be nice to see Dragons make a kind of cameo appearence, but not play a central role. That would duly add to their mystery and intrigue while not pandering to the overused fantasy tropes. Also: I will be absolutely overjoyed the first time i see a Sload.