The rest of the time I play on default.
I have no interest whatsoever in making the game 'harder' or 'more challenging' in those aspects the slider influences, thats really not why I play games.
Oh I wish difficulty sliders influenced puzzles and riddles and such. If there existed a slider that could make the puzzles fiendishly difficult, I would put it all the way to the right.