In a more linear game I might, but given the nature of Skyrim's open world, no.
This is my general attitude. I play most games on Normal, and they're usually fine. In the rare game where I'm no longer enjoying myself, and can't really go somewhere else and try a different path, tho? Yeah, I'll adjust the diff.
Examples - Half Life 2, the prison scene where you defend the alcove with the wimpy turrets. After a bunch of tries, I turned it down. And at that point - I wasn't interested in challenge as much as I was interested in the story. So I left it lower. (I forget if I turned it back up for the Episodes.)
Dragon Age was another - PC version's higher difficulty, combined with a less-than-ideal party (two rogues, healer, tank. Yes, I played "Dragon Mage":Origins with no mages...) made for a horrible experience. I initially just turned it down for the boss fights, but then it started throwing me into automatic fights on the road against 15+ enemies. Difficulty went down and stayed down. (And again, there was also the fact that I didn't enjoy the combat. It got amazingly tedious, very quickly. But the story was interesting.)
edit: I've never adjusted MW/OB/FO3 away from Normal.