This is a hard one. Well at least in Oblivion it was and I really doubt it got easier in skyrim. One of the issues in Oblivion was that several of the flags that told an NPC to follow the player was just an actor that was FOLLOWING the player. That COULD mean an ambushing enemy actor or spy not a friend.
Several different strategies were tried but none of them worked in every situation except for the ones that required the player to "mark" an actor as a "companion". For example I made such a mod where you gave a "friendship ring" to the actor. This ring could be detected by any other mod with only 6 lines of code to identify the actor as a companion using a system that did NOT require dependency on my mod (but it did require OBSE).
So the real question is going to be what do YOU define as a follower. Then see what you can do to detect the conditions you define as a follower.
A good start is being in the same faction, not hostile to the player and flagged to follow.
But then if it were me I would use my friendship ring idea instead.