I guess it depends on which side of the fence you are on. But i think the loyalist will be the ones still loyal to the empire and the others will be branded as the rebels or something.
Well, I guess you're technically a loyalist either way, your just loyal to a different side, like the "rebels" are loyal to Skyrim, and the "loyalists" are loyal to the ever crumbling and fragile so called "empire"... (which really only controls Cyrodiil and [sortof] Skyrim, right?)