That's the great thing about roleplaying... you can make just about anything work.
As a child you found you had a natural talent for magic, and became a priest schooled in restoration, and when the Great War arose you traveled with your kinsmen to aid them in battle... but your unit was ambushed by a Thalmor detachment, and slaughtered to almost a man while your restoration powers could do nothing to save them. The following week the White-Gold Concordant ended the war, but you vowed never again to be helpless in the face of the enemy, and returned to Skyrim to learn the ways of destruction & conjuration.
Or...
Your father was a sea-captain, a merchant trader who took you along on his voyages as a child... until a massive storm capsized the vessel, and you washed up on the shores of Highrock. A Breton noblewoman riding along the beach found you and took pity on this orphan child, and took you in, giving you the traditional Breton education in magic. At the age of manhood, you decided to leave your adopted home to return to your ancestral home, hearing about the Civil War erupting in Skyrim... until you were captured by Imperials crossing the border...
Or...
As the nephew of a former court mage of Whiterun, you were trained in the arcane arts to one day take his place... but his sympathies for the Stormcloaks eventually led to him being cast out of the Jarl's house, and replaced with a more "Imperial-friendly" mage. You set upon a journey with your uncle- not knowing where he was going- until he arrived at a Stormcloak camp. Not wanting to take a side yet, you turned to head back to Whiterun... and ran smack into an Imperial patrol that was encircling the rebel camp...
...Whatever you can think of. Have fun with it. And if it doesn't fit the lore exactly... who cares?