If I buy a horse for myself, what happens to my follower? Can I buy a horse for him, too and will he actually be able to ride it?

Nope, but he will follow you, and if you ride too far from him he will just "spawn" closer in a few sec or so, so it is generally no problem

they manage to follow on foot somehow.

Gah! This is the same problem that Oblivion had. You either have a horse or you have a companion =/