I mean lore wise? Did something happen to them?
Pretty much what hagrid said. That and the Betrayed already own skyrims caves and caverns.
I don't think Goblins are native to Skyrim as said already. Besides, I believe the Falmer make for a more interesting replacement.
There's goblins in Skyrim! At least in my game, cause I use immersive creatures mod !
There's no comparison. The Falmer aren't just a bunch of dumb creatures, there is much history behind them which cannot be said about the likes of goblins.
Rieklings have much more in common with those.
TES "goblins" are not universal in their distribution, though every place seems to have some sort of equivalent. They were in Arena, but not in Daggerfall (and therefore, I presume, not native to High Rock or Hammerfell).
They also weren't in Morrowind, except in Mournhold, and there only because Helseth had some high-elves building him a goblin army--they were imported. Solstheim didn't have proper goblins, but it did have "reicklings", which are very goblinoid in appearance. Mainland Skyrim also has a goblin replacement in the Falmer, and indeed, it was said in Solstheim that some Nords refer to the Reicklings as "Falmer".
Personally, I believe the predictor of the presence of goblins is the historical distribution of elves. Goblins and goblinkind are degenerated elves, an each area has different kinds.
High Rock has Orcs, referred to as "Orsimer", former followers of Trinimac who were changed by the daedra prince Boethiah.
Skyrim has Falmer, still referred to by their original name, except in Solstheim where their relatives are called Reicklings.
Morrowind is inhabited solely by an elvish derivation: the Dunmer, or "cursed ones", who were originally called "Chimer", but changed through interaction with the Heart by the Tribunal (and before maintained by a relationship with the Daedra).
Cyrodil was originally inhabited by Ayleids, called "Heartland High Elves", who don't seem to exist any more (excepting a few long-lived individuals), but there are Goblins. There are also Goblins in Summerset, where they are sometimes used as soldiers. I suspect "goblins" as we know them are what happens when high elf lineage degenerates.
I don't know what lurks in the forests of the Bosmer... and I also have no idea about their origins, other than the Khajit notion that the the two have a common origin, but while the Khajit forms were stabilized by Azura in conjunction with the remains of Lorkhan (the moons), Wood Elf forms were stabilized by Yffre. So its possible Valenwood has no native goblinoid population, either.
To summarize, my theory is that, wherever there are "meric" populations that have been stabilized through interaction with some sort of a Padhomeic divine (whether the Tribunal, Trinimac/Malacath, Yffire, or Azura and Lorkhan... or possibly even the Heart and the Numidium, which might also explain the absence of a Dwemer-derived Vvardenfell goblinoid population), there are no goblinoids. Wherever the elvish population found no daedric or mundic root (which I speculate include the Altmer, the Ayleids, and the Falmer), goblinism afflicts at least a portion of the population. Therefore, one should not expect to find "goblins" outside areas where Altmer populations once ruled. Skyrim was Falmer territory, and so that is what you find there, instead of goblins.
Men, on the other hand, have always been more rooted in the Mundus, accepting Lorkhan/Shor/Sheor/whoever as a primary deity, and therefore not suffering goblinification.
The Goblins in Oblivion also seem to have some connection to feathers as did the Aylieds.
As for the Falmer, Gelebor says that they've been improving in recent centuries which is odd if all Goblins are in fact degenerated elves. Blindness and enslavement to the Dwemer would have made it rather hard for them to stick to whatever breeding measures the Altmer and presumably Ayleids and Falmer used to stop themselves from degenerating.
If towers are somehow involved in stopping them from degenerating it would explain why the Ayleids and Falmer degenerated so far and why the Altmer have gotten so desperate now that Crystal like law has been destroyed.
The Ayleids practically disappeared within a few centuries of losing control of White Gold Tower. At least those that weren't long lived or intermarried with the Bosmer or Direnni.
The Falmer also completely disappeared within a few centuries of losing control of whatever their tower was either Snow-Throat, the college of winterhold or whatever it might have been except for those who where long lived or vampires or remained at the chantry. The War of the crag also happened a few generations after the Dwemer first enslaved them likely after they started to drastically degrade and they could no longer control them. The Dwemer seem to have only intended for them to be blind for whatever it was they used them for.
I should mention however that there's also mention of Goblins existing before the Ayleids lost control of WGT or the Falmer lost Skyrim the Redguard also could also Apparently Identify goblins when they first landed in Hammerfell so that implies that there were also Goblins on Yokuda so wether they're actually degenerated elves or not they're at least found anywhere the Altmer, Ayleids and Left-handed elves were found.
There were also goblins on islands close to Akavir which the Tsaesci although there's no evidence of how they ended up there currently. Possibly goblin slaves where on board the ship of one of the other captains mentioned in Father of the Niben. Or that's where the ship that wen't south ended up if it missed Pyandonea and the ships crew ended up devolving after being stranded.
http://www.uesp.net/wiki/Lore:Father_of_the_Niben
I've always had the belief that Goblins are related to Malacath, similar to how Orcs are related to him.
I seem to remember some source saying that they were simply big goblins as well as a something involving the Redguard fighting both Orcs and goblins pouring out of portals when they first arrived in Hammerfall.
http://www.uesp.net/wiki/Lore:Pocket_Guide_to_the_Empire,_3rd_Edition/Hammerfell
That's the only mention I can find of them fighting Goblins and Orcs when they first arrived but I swear I read something about orcs, goblins and portals in the past. Maybe it was just in one of my many annoying dreams that I always wake up from more tired then when I went to sleep
http://www.uesp.net/wiki/Lore:Pocket_Guide_to_the_Empire,_1st_Edition/The_Wild_Region
http://www.uesp.net/wiki/Lore:Goblin-ken
There's sources from when Orcs were largely considered nothing but bigger slightly more intelligent goblins.
GOBLINS...
stronger than Dremoras, Daedroths, Xivilai's, minotaur, ogres and trolls
FRIGGING GOBLINS!
Nah. The Ogres and Minotaur Lords are stronger overall. Goblin Warlords have a large health-pool, but their melee damage is low in comparison to the other boss enemies like the Ogre and Xivilai.
Oh and, Gloom Wraiths.
Ogres would be more difficult if they didn't waste precious seconds taunting you every time. Minotaurs are kinda easy to dodge although their headbutt is pretty damn powerful and annoying. The thing that makes Goblins so tough and worse than those two is their relentless attacking, ultra fast healing and their guerrilla tactics.
Oh I forgot about Gloom Wraiths...
Gloom Wraiths -> http://www.gifbin.com/f/982209