Goblins where also never a great and ancient civilization at one point. Unless that one theory of Cyrodilic Goblins being devolved Ayleids do to their loss of White-Gold is true.
At the Very least the Nords would be hard pressed to defend Markarth if the Falmer decided to raid it. They have the advantage of the Tunnels and Dwarven ruins to freely move all over skyrim. They would pretty much be able to raid the surface with impunity well the Nords are busy fighting the Thalmor or anyone else, they wouldn't have the manpower necessary to contain them in all the caves and ruins let alone go in and deal with them decisively they also have the advantage of having almost 4000 years to adapt to the ruins and caverns.
The Falmer are more of a threat then people like to admit as for their Intelligence it's hard to Judge. Gelebor has noted that they have Increased noticeably in intelligence since they first sacked the chantry and that there may still be hope for them yet and he would be a pretty good judge of whether they could ever come close to their old selves or a comparable level. They are also advanced enough to raise chaurus for various uses perform Alchemy, craft weapons and heavy suits of armour and they still have some degree of Magical prowess although certainly nothing on the scale of the other races any more at least no t that we've seen.
Heh this has gotten really of topic of whether the Empire should still be called an Empire.
So on the original topic It controls it's hope province and at least western skyrim and is ruled by an Emperor so it qualifies for being referred to as an Empire. But I don't think it deservers to be referred to as The Empire. It's fallen a long way from the Height of the Septim Empire or the First Empire for that matter. Its certainly lost all of the glory of past Incarnations.