The Reachmen KILL people. Their rule wasn't peaceful, the writer of bear of markarth is an imperial ass kisser.
Thus was brokered to the witch: his heart, his will, his humanity. From that day forth, his was a spirit of vengeance, pitiless and beyond remorse. The rebels grew in strength and numbers, and none could stand against them. Faolan's eyes burned coldly in those days, black opals reflecting a mind not entirely his own. Two years passed, and the foreigners were all but driven from the Reach.
-http://www.imperial-library.info/content/skyrim-legend-red-eagle
THIS is what forsworn are. They aren't a kind people. The ones in the wild attack anyone without prejudice. They murder people for the Hagraven's rituals.
People seem to be blowing the racism thing way out of proportion. The forsworn briarhearts aren't human anymore. Most elves have no issues with stormcloaks save the grey quarter elves. One of them even mentions that they aren't getting along because the Dunmer don't want to fit in. http://imperial-library.info/content/skyrim-decree-monument They don't pay tithes to the Jarls, or even owe allegiance to them.
2 drunks in windhelm(Who aren't even a part of the stormcloak forces) is a terrible way to judge an entire cause.