I do not get any kind of quest in my journal from this.
He just challenges me at random.
Is this supposed to have a journal entry associated?
It's supposed to pop up under the misc. quest section of your journal after he's issued you the challenge. It's something along the lines of defeat the dueling wizard.
I had to look this up. In my 100's of hours of gameplay (including my current Mage with 100 hours and 100 skill in every Magic school), I have never once encountered this quest. Weird.
Probably completely coincidental but I never see him until after I read "The Apprentice's Assistant" which details how to do a 'proper' mage duel.