So as AFAIK in Skyrim khajiits are not allowed in cities. Is there any explanation in lore (not just because of gameplay) that can rationalize why, even if you don't do the MQ, the player can enter cities as a khajiit in Skyrim?

It's the Khajiit caravans that aren't welcome, not the race in general.

One Khajiit won't cause problems. Multiple Khajiit can.

Especially when the Khajiit we see in Skyrim tend to... "find" their products for sale. In this case, it's more about preventing a pack of possible thieves from entering the city and causing a massive crime spree. A single person can't do that much (unless they're the PC,) so the city guard doesn't worry about singular instances. However, a mass of people?

There are a handful of instances IRL known as "Mass Theft." In these instances, many people (in some cases numbering more than 50 or so individuals) basically storm a store, take everything they can, and then leave. The hope is that the police will be unable to track everyone down. By preventing the Khajiit Caravans from entering the cities in the first place, the Guard doesn't have to worry about this event, nor suddenly not having enough room in the hold's dungeons for everyone.

Oh ok, I was confused about that

Not really. You can kind of explain Whiterun since you survived a dragon attack and were trying to warn the Jarl, and that the people of Riverwood specifically asked you to ask the Jarl for re-enforcements. Riften may be the other sole exception since it's corrupt as [VEHK].

Yet we never see any Khajiit in the cities (except a Thalmor assassin in Riften, who only goes there at a specific time under orders from Elenwen). I mean heck, if you can find an multiple Altmer in Windhelm, you'd think you'd be able to see at least one Khajiit in Solitude. It's not like all NPC Khajiit in the game are from the caravans.