I don't either. It's amazing how much larger Skyrim feels to me when I'm not constantly looking at a scale model of the place. It also tends to feel - dare I use the word, Renee? - more immersive to me. I feel more connected to the landscape and to my character when I am not constantly exiting the game world to look at a map.
In fact, I became so fond of the idea that I've begun doing the same thing in Oblivion and Fallout 3. Nowadays I only use a map when 1) my character might plausibly be lost and 2) when my character might plausibly have a map in her possession. If either of those do not apply I do not use a map.
The only problem is that there are a few quests in all of Bethesda's post-Morrowind games - such as A New Order in Dawnguard (in which Gunmar might be in any one of fifteen random locations) - that require us to use a map, not for roleplaying reasons but because developers force us, the player, to use it.