Yeah, spend a little time in Cidhna Mine and see if you still like the people. I've killed all the non essential Silverbloods and numerous guards. Woe unto the guard who wanders into the shadows in Markarth. My home there is crammed full of guard helmets as souvenirs.
I'm on Silverbloods side with that one. The Forsworn is an old tradition that needs to end. Killing their leader was the best way to end that. Actually, the alchemist in Markarth will tell you same thing basicly.
So... Yeah. I understand it all. I got thrown into the mine because the Silverbloods thought I would cause trouble but instead, I helped them and saved a ton of trouble for them.
I also like Silverbloods because they are on the Stormcloaks Side.