So I killed him, and found that right after the battle my personality lowered from thirty something clear down to six. He attacks you first, it's not like I did anything wrong. Why is this happening? Really kills the joy of getting that armor...

It happened to me once. My opponent had Chastening staff or something which damages personality. You can restore it at imperial altar or temple.

Ah, ok. Cool. Good to know it'll be an easy fix.