If you have any interest in a "sword and sorcery" type setting with a deep (and albeit very convoluted) history and lore, then I would definitely get the game. Also, expect to invest a serious amount of personal time before you see a satisfactory return in understanding of the world of Nirn; It requires a tad bit more of you being involved as it's much less about shooting your way through ghouls and drinking from toilets than pausing to understand what it is you're doing and why. Fallout is a base and profane survival game in a post-apocalyptic-alternate-earth , TES not so much... It invites you to come and live as you like and be a part of the world.
Although Oblivion was the 4th installment of the series (not including two spin-offs) it will definitely give you a better perspective on exactly what the series is about, and help you to appreciate Skyrim tenfold.
why not get all the DLC (GOTY)
GOTY is KOTN and SI. Nothing more. The remaining DLC is only available through KOTN discs or from Bethesda's store (currently not online).