So basically whenever I do the dungeon Fort Vlastarus and then go talk to people, most of them (except guards) tell me to "get away from me" or "you'll not have any of my blood!"

Sometimes the guards tell me I look sickly even though I don't have a disease (and I definitely don't have vampirism). So far I have found that this only happens when I complete Fort Vlastarus, even though I cure porphyric hemophilia when I get it. What the heck is going on?