Throughout TES, Orsinium has been in the Wrothgarian Mountains near Wayrest. However, one of Skyrim's loading screens describes it as "a small kingdom in the mountains between Skyrim and High Rock." Is there any other record of this change in location? Did Orsinium just get retconned? Or did the guy who wrote the loading screens just make a mistake?

Probably a mistake.

It's kind of central High Rock, isn't it?

It's not *really* a mistake, just hazy terms. The statement could technically be construed as accurate in accord with Orsinium's established location.

I believe that loading screen actually says it's between Hammerfell and High Rock, though I may be wrong. It also goes on to mention that it's been destroyed and rebuilt many times, so if it were moved, that would cover it.

Orsinium as been conquered, razed, and rebuilt over and over again. It might've changed location at one point or another.

It was sacked again in the early fourth era. Then they moved. You can visit several orc encampments if you do some quests for a NPC orc. Apparently skyrim was big into taking hiuddled masses ~200 years ago.