I dunno, I think we would have also heard if the Altmer had a vast horde of goblins ready to be mobilized. Admittedly, I am not familer with the ins and outs of Tiber conquering the isles with numidium, and even though I never thought to ask, I was operating under the assumption that Tiber would have crushed any forces mobilized by the dominion, including Goblins, thus explaining we we don't hear about vast amounts of Goblins freed, co-existing with Altmer, or vast amounts of Goblins waiting in the ranks. Correct me if I'm wrong, please, I'm only here to learn.
Oh, OK, yeah. I get where you're coming from there. I also doubt that the Altmer had some vast horde of goblins, ready to strike at any moment and overwhelm the rest of Tamriel. The Empire wasn't exclusively run by total idiots after all. They probably restricted the armed forces of all the provinces. But, for example, there were still Knights in the Illiac bay, and those knights had warhorses.
SilentColosus says NPCs confirmed Altmer Goblin-Rearing, so I think that settles that.