As was said above, it is in their nature, they are Pariah. The fact that they worship Malacath does not help their cause. And them attacking and ransacking Breton villages and cities did not help in their past. More in current eras, after they helped the empire they were helped and protected. But still a clan of people banding together and taking over your province (High Rock and Hammerfel) is bound to be met with opposition from these highly territorial people (Bretons and Redguards).