Did you check your journal if you still have the notes related to it?
If it isn't, you failed the quest but the follower stayed "with you". Which may cause a problem when trying to speak with the quest giver. (If the quest is over, he'd normally talk to you again normally)
So if it's the case, wait a few days and the follower should eventually disappear.
Or you can pretend you took the merchant's wife.
This reminds me when I was working for the same merchant for a couple of quests. Out of the 6-10 quests I did, I had to save 3 of his cousins from the Dark Brotherhood or the Thieves Guild. Seems like that guy had a shady business going on.