If I stumble upon a mountain troll and I don't want to fight them, I'll cast invisibility and go on with my day. My follower, on the other hand, will probably be ripped to shreds.
Why not either let me cast invisibility on them or let me teach them the spell so they can cast it when appropriate?
What do you think?