I cannot answer that one as I haven't observed it. As far as I can recall, none of my followers have keys to player home(s). It may be that:
1. Because they are followers, they get ported to your interior location without ever having to pass through actual doors, or
2. Once the door is opened, any NPC who can open a door can open the player home door.
To test this, you could try a couple things in game:
A. Make sure a follower does not have a key in their inventory to your home
B. From outside, tell them to "use" the door to your home and see what happens.
I've noticed that some NPCs who have scripted interactions with the player can result in them "following" the player out of an interior cell in order to finish giving you their lines. It may also work going IN to interior cells, but I don't have a clear recollection of that.
You could use the console to create a need for a courier, then spawn the courier right next to you, and when he starts yacking at you, tab out and go into yours house . . . actually courier might not work, it might need to be some other NPC, but you get the picture.
There are probably a few experts on how all of this is programmed who will post in a bit and clarify all of this and save you the guess work.