I think genre (fantasy vs sci-fi or post-apoc) is not all that relevant to how well Fallout 4 will do versus Skyrim. There is an ebb and flow to which genre is most popular at the moment, but quality content finds a market regardless.
I loved D&D and Conan as a kid, but I got sick of the whole fantasy thing as I grew up. Particularly well done movies like the LOTR series were still enjoyable, but in gaming, wizards and elves just didn't do it for me. I loved Fallout 3, but wasn't too interested in TES. I followed Skyrim's development out of interest in BGS--I wanted to get an idea how the engine was improving for the inevitable Fallout 4. I ended up buying Skyrim on a whim at release, and proceeded to lose a few hundred hours in it. I suspect that a lot of TES players came to Fallout in the same way.
With each new release, BGS does a great job of pulling in new fans while retaining their fan base who've become addicted to their style of open world game (regardless of genre). I'm confident that the trend of ever-more-successful games will continue with Fallout 4.