http://uesp.net/wiki/Skyrim:Wizard_Duel

I got the challenge in Whiterun. I excepted the challenged then calmed him down and continued on my way as I was not prepared at that time. It has been several days, I am ready yet the Kill Challenger quest has no arrow to point even when it is turned on.

"If you enter another cell while fighting against the Challenger, you will lose their location and be unable to finish the mini-quest."

Can I get him back to finish the quest?

If the UESP says you can't, then no. Not possible. Unless you're on PC.