First let me say that I've never used a follower before. That being said, how much of an impact on skill progression does using a follower have since they will be doing damage and taking damage that you would otherwise be doing or taking? Since doing damage increases your weapon skills and taking damage increases your armor skills.

Good question - I ran into this issue when I got Lydia during my first play-through. She would charge in and kill everybody so no progression on armor and weapon skills there. Of course, you can improve your archery...

I think you have a couple good alternative:

* Only use followers strategically (have them wait outside a ruin and use them as a mule for loot)

* Get good at conjuring so you can summon atronachs as needed (very, very fun in my opinion)

Good luck!