But I would have to adjust it every time I go into a cave. I do not want the outside world to be darker than it should be.
Not really. As these are consoles*, and most folks flatscreens are overly bright, dropping the Gamma so it is almost dark at night/in caves still gives a fairly bright daytime. You may not be able to get it as dark as you like depending on your TV/Room set up, but you can get close. At least on past games.
*Most people don't play a console on a PC Monitor. Which are even worse for being overly bright our of the box.