I would never followers or staffs in this game and I always play on Master. Why do level ups come so quickly? is it the difficulty i play on, or does it not matter?

On my new playthrough i just started Im having Feindal do a lot of the combat and using staffs to kill enemies. I also try to use the environment in fights, like shooting fire pots in the ceiling with my bow. I try not to pick too many locks, make potions, or enchant gear. I increase my speech by selling stuff, that is pretty unavoidable, oh and I crafted Feindal some ballin' leather armor. So far so good.

Does anyone else try to minimalize skill usage on Master difficulty so your not leveling up so quickly and fight tough enemies early? Does anyone have any tips that'll help me in this playstyle?

Nope. I just have one because I like to.

I think many people use followers on Master.

Sounds like you also need to raise your other skills as well.