I dunno, your definition of "fantasy" must be pretty fanatical if The Elder Scrolls doesn't fall under its umbrella. Magic, undead, drastically different races that somehow still exist on the same planet, a crazy number of conjured-up creatures and other animals that don't exist in reality, "deadric lords" that can intervene in everyday life, separate planes of existence (some of which can be traveled to and from) - it's even got a unicorn.
I think he was confusing "Genre" with "Thematic Style".
"Fantasy" is a thematic style which is usually characterized by magic, supernatural beings, and epic storytelling. Oblivion's thematic style is quite solidly "fantasy".
RPG is a genre which is characterized by controlling a single character or small party of characters, controlled/affected statistical growth, player choices, and long story lines. Oblivion's genre is a hybrid between this and Action/FPS.