I've already started a game as an Imperial but now I feel that this isn't right because lore says that Dragonborns are Nords. Should I start over for sake of lore accuracy??

No. Dragonborn do not have to be Nords at all. It is the soul that matters not the blood. Play whatever you want.