First off, don't use a follower. They inevitably charge into the attack and bring the whole lot of the cultists down on you.
From what I recall, I too was astonished at how potent they were and got laid out several times.
In the end, I went alone, leaving the follower at home. I managed to work my way around the back of the camp keeping to the right while sneaking and cutting right up the rocks as soon as I was able to. I think my sneak was up to 4/4 by then and the skill around the 80 mark. It's possible to scramble up the rocks behind them. It sort of plateaus out above the shrine area and you can take out the cultists near the altar from there before sneaking down and getting the camp cultists from there. This method often works because these guys at the top are the lookouts for the others. I took them out using sneak attacks with a legendary ebony bow that was enchanted with fire damage: I can't recall what damage the bow had. I worked from back to front of the camp waiting to make sure the others stopped looking for me with each strike. It would help if you have bonuses like x2 or x3 damage for sneak attacks. Invisibility may help but I didn't have it at the time so it's possible without it.
Even when you do kill them, you can't get rid of the quest without leading a follower there. I tried leading random bandits up the hill but they wouldn't even follow as far as the altar. In the end I picked a mercenary who seemed to have the worst attitude problem.
If the worst comes to the worst you could always drop the game difficulty level. I did it on Adept and was sorely tempted to chicken out and go to novice for this one. I only didn't because I remembered the Azura's star business where I couldn't manage it at all, saved up heaps of potions, waited 'til I could fashion armour like the Fort Knox vault, weapons that would put atom bombs to shame, dropped to novice... and found it too easy!
P.S. I'm pretty sure it is possible to do that kind of damage with one strike. I never checked what quality of weapons they were using but I have a dragonbone sword that does 350 damage and I have all the 1-handed perks. Maybe these npc's also have perks similar to bone-breaker?