If you talk to someone outside and there hap way through there sentence I think they go in your house but then leave
EDIT: Don't worry the bounty for physical abuse in skyrim is 40 gold I believe, I'm sure you can cough up the dosh.
The Champion of Azura doesn't talk to guards, they talk to me!

(but yeah, he was probably blabbin and followed me in)
The rage I felt at his intrusion, I didn't want to hit him. I wanted to kill the chump

But I don't kill good NPCs in game so I can't

But yeah, I think he left