Hm, I'll answer this in the persona of my character...
"Hm, Allegiance is a strong word...I have many teachers, many friends and allies, and a few...convenient relations...But with exception to when I visit a shrine to the Nine...Yes I said Nine...I refuse to bend my knee to anyone (Hermaeus Mora never seems interested in fealty anyway, though he DOES charge steeply for overdue books, you know). When we took WIndhelm and made a mess of Ulfric on the floor, I hung around just long enough to be polite before strolling out of that wretch of a city, until I found a river to throw my Imperial armor and sword in, and let my servitude to Tullius wash away and washed my hands.
"Granted, once Rikke tried to persuade me to return to Solitude with her for some bit of Imperial business, and I actually had to threaten her with being shouted off High Hrothgar if she didn't leave me be. My time with the Empire was done, because I said it was done. I'm the Dragonborn. Who's to tell me otherwise? I only really go involved in the civil war because I hate the Nords (Neandethals) and hate Tullius a little less. But now I can concentrate of my studies and research."