Not to mention the consignment of Dunmer to the grey quarter, the Argonians - who, by the way, are working and don't complain/whine all that much, they are completely denied from entering the city at all! So that's one massive, massive 'against' to the Stormcloaks.
Another issue, you gain the knowledge that the Thalmor consider Ulfric - due to his rebellion - an asset of sorts. Given that, obviously, the Thalmor must be opposed, there's another strike for Ulfric, who is inadvertently aiding the real baddies.
Now, the rulers supporting the Sotrmcloaks... Some are outright bigots (e.g. Winterhold, the jarl is against all magic even though the college is the only thing left for his city, or Dawnstar, where the jarl is persecuting two of his inhabitants just becuase they were, in the past, working for the legion.) Also, he is on the same side as the Silver-Blood family, who are just plain bastards. Another strike for Ulfric.
Additionally, if you read the in-game book "The Bear of Markarth", you know that Ulfric, on retaking the city from Reachmen, slaughtered not only the men, but also the women and children, even after they had surrendered! Another strike...
Furthermore - speaking to Solitude's courtpeople, you know that Torygg would actually have supported Ulfric's claim of disbanding from the empire - but Ulfric was either too powerhungry for the throne himself, or too blind to politics, that he went and challenged Torygg to a duel, where he had a magical advantage. I don't know if that would count as a fair duel or not - seems like an abuse of the Voice and the way of Greybeards, but then again Talos also used the Voice in battle, so.. Idk. I can't say this gives any positive light on Ulfric.
So there's all that. I can't really think of an argument that would convince my character - who is Argonian, just to recap - to side with Ulfric.