Of course they'd still have a capital city of Orisinium! They are proud of their heritage, but I don't know many orcs that would refer to themselves as Orsimer.
If you still aren't swayed after this, I give up. Here are some quotes from "A Pocket Guide to the Empire," the actual little book that came with the Oblivion collector's edition:
"If the Myth of Mauloch is to be believed, the Orcs or Orsimer had been pariah of the land for two hundred years, and if it is not true their destitution had been even longer."
"The Orsimer exodus was long and particularly deleterious to the character of its people." (referring to the time after the siege, AKA when they were already "orcs")
And throughout their
entire section of Orsinium they refer to the Orcs as both "Orcs" and "Orsimer."