I have not even engaged in a single quest to join either the Imperials or the Stormcloaks. I have cleared out forts in the wilderness such as the one west of Whiterun of the bandits in them.

I came back later on and it was filled with Imperials. I also went past intro towns and stuff and found that bandits now took other buildings over.

Is this all random or specific to these particular forts over time?

I had a Fort of Bandits go Imperial .. I left to sell off some stuff and when I came back to finish looting it was taken over.

Yeah, but is it random? Can that same fort in another game be Stormcloaks instead?

Yeah, that particular fort will become Stormcloak if you're aligned with them. You get a quest there from one of the factions, and the questgiver mentions that, at least.

Thats my question though. I didnt do anything for it to become imperial. Can it become stormcloak like it did imperial with no input from me? As in random? Or do these specific forts go specific routes unless you do something yourself as in questlines?

I was wondering the same thing this morning. I too haven't engaged in that story arc and all of the forts that I have cleared out have been re-occupied by one faction or the other. It sort of makes sense though; a large empty fort just waiting to be used as a fort by soldiers.

I haven't joined either side yet.. so is it just random?