I cannot remember where, who, or when.

Try to reload the oldest save you have, then try to deduce the most recent save where you have lost him and reload that.

or, you could use console commands to teleport them to you

Kingoscruff

console commands sound good. Where would I find them?

danc

press the ` ( or the ~ button) to get the console up and http://elderscrolls.wikia.com/wiki/Console_Commands_(Skyrim)http://elderscrolls.wikia.com/wiki/Console_Commands_(Skyrim)http://elderscrolls.wikia.com/wiki/Console_Commands_(Skyrim) and use the uesp http://en.uesp.net/wiki/Main_Page

In order to use the console commands, I think we will first need to determine which follower you have. Where did you first pickup this follower (town / city, inn, etc)?

Maybe that is why they left.

Assuming you're on PC open the console and type

SQV DialogueFollower

It should list a number of variables as well as the quest alias that contains your follower, their name and their i.d.

once you know the i.d. you can type

prid

moveto player