When they attack you, unsheathe your weapon, then sheathe it again...try again if it doesn't work the first time. If they still attack...well that means you probably have declined to go to jail earlier. They will hunt you down for good now.
You could also try this: Leave the town, go far away, like Whiterun. Then wait for a few days. Between 3 and 7. Then see if they still attack you in Markarth. This method works for instance when you attacked a legion patrol to free a prisoner, so the legion will attack you at their campsites. But if you wait for a few days they "forget" and are fiendly again.

It's not bug, it's a feature. And it should work for all factions.