Hammerfell didn't begin to fight for themselves in earnest until several years into the war, as the Forbares and Crowns had been too busy attacking eachother and holding back out of fear the other would gain the advantage. It was really just the A'lkir warriors and the Legion, and since the Imperial Legion discharges formed the backbone of their armies during the 5 year extension it was hardly alone. The battle of the Red Ring and the preceding 4 years also crippled the Dominions strength, and logical military precautions would have the Dominion deploying a force South of Cyrodil to counter any invasion during their assault on Hammerfell.
The Empire isn't weak, it's one of the two big powers on Nirn, and there are a lot more than just High Elves in the Dominion.