Yeah,not being a nord dragonborn seemed lore breaking to me,even as a khajiit player.
There are a few Akaviri races that carries association with dragons, the golden snakemen in particular, along with the tiger people and their Tiger Dragon King, Tosh Raka. And the snakemen had a lot of influence on the Imperial culture under the Reman Empire. And there is the whole thing about Alessia and her pact with the Dragon god Akatosh.
Prior to Skyrim(game), the Imperials had more to do with the Dragons than the Nords did.
there are plenty of Dragonborn that were not Nords...
Imperials and Bretons have been Dragonborn (the Septim Dynasty)
and we know there was at least one Altmer Dragonborn (good ole Mankar Camoran)..
So no, Dragonborn are not limited to any one race