Though I know each of the provinces aren't always inspired by real world cultures. I was wondering if the Khajiit were loosely based off of Middle East/Eygpt? Mostly Their accents (though I really wouldn't know) and the fact that they live in a desert/arid like climate. And the cats remind me of Eygpt alot.

Easiest way to tell is look at their architecture.

Of which we don't have a clue.
Personally, though, I think Elsweyr is inspired by ancient Persia.

Going by their names and what we know of their language, I'd say medieval Middle East / Iran / India - the three were very closely connected at the time anyway, so there was a lot of cross-cultural mixing. They have a strong tiger-ish aspect to them, as well, which helps to put them in the latter, and they also seem to be connected (vaguely, at least) with the D&D interpretation of the rakshasa from South Asian mythology.