So far, I haven't touched it. But I was wondering if there will be any impact making the game too easy or too hard.

On the wiki they advice leveling the skills might benefit with a higher difficulty, because the fights take longer.

My main question, will there be a difference in drops?

here's what i did ad was a quick and ez way to see how difficulty impacts gameplay: run through the tutorial twice with the same build on normal and the highest difficulty;)

No, it affects nothing but the relative damage given/taken in combat. When you move the slider to the right, you decrease the damage you cause to enemies, and increase the damage you take when they hit you. And that's the only thing it does.