Imperial Soldiers are one thing, I'm more worried about the nobles of Cyrodiil. There's little doubt in my mind the Thalmor are lining their pockets with gold for their affection.
I doubt those nobles have much pull on The Emporer's actions, though.
EDIT: My main reason for hating Ulfric is actually because of that speech he makes when you first enter his hall; he talks about how hes fighting for those who have died and blah blah blah, but all he is doing is getting more people killed. I found it very hard to take him seriously after that blatant hypocrisy.