I do the same thing, Ive roleplayed Im the same character in Daggerfall, Morrowind and oblivion.
The way I solve it is reincarnation. My hero appears to be (much to his dislike) a soul the gods call upon when the world needs saving. He believes the gods are playing a quiet joke on him, or maybe its a punishment, because he eventually remembers all his past lives every time.
In the case of oblivion, it wasnt until he was in jail over a
complete misunderstanding about a horse that the memories of his previous live(s) started to seep through.
This memory helps him to know what to focus on in this life. Just because he was a master of destruction last time does not mean his new incarnation is from the start, but he knows he likes destruction and will focus on training it.
People sometimes say this is impossible, as the events of those three games happened so close together, Id be an infant in oblivion if I reincarnated just after Morrowind.
My answer to that is: Who said reincarnation has to be temporally linear? So I reincarnate a few years before I die, my own personal timeline is still a linear progression, Morrowind after Daggerfall before oblivion, its just that compared to the normal timeline I appear to jump around a bit.
Technically that could mean that at some point in time three seperate versions of me were alive, and if they ever met up probably the universe would fold up like a chrysantemum, but Im an Altmer and know better than to do foolish things like that.
It is my characters fondest wish that at some point he could have a life of quiet leasure. Reincarnate as a wealthy noble in a stable period in history perhaps. Spending his days reading books, collecting butterflies and never ever worrying about global events.