... Or
HIt em with a hammer...
REPEATEDLY
...Which actually isn't a half bad idea. That, or a claymore, whichever you're more proficient with. With some fancy footwork, you can well out-range Agarmir with a warhammer or claymore. Neither he nor his buddy use any magic, IIRC.
Another thing that can be done in preparation for the fight is to buy up some healing and respite potions. Agarmir's sword, Debaser, is rough because it lowers your base Health and Fatigue by draining your stats (hence the name). If you're getting clobbered, hit the menu and quaff a few potions to stay in the fight.
Speaking of potions, how about poisons? Even if you're a novice Alchemist, you can make a couple poisons to wear him down. Rat Meat + Bonemeal will make a Damage Fatigue poison to use on him. Nightshade + Wisp Stalk will make a Damage Health poison.