I just got done with the first vampire quest that was given to me by the leader of the Dawnguard. It's the one where you have to bring the girl back to her castle and are given the choice to become a vampire or not.

I chose not to become a vampire and was banished from the castle.

Well now I'm starting to get the vampires abilities minus actually changing into one.

I restarted that quest 2 times

Is that a glitch or is it actually part of the story?

Sounds like a glitch. Are you on console or PC? Regular Skyrim or Special Edition?

You might have been infected with vampirism in Dimhollow Crypt. Check your active effects tab and see if it says you have Sanguinare Vampiris. If you've already turned into a vampire Fallion in Morthal can cure you in exchange for a Black Soul Gem, otherwise a cure disease potion or activating a shrine will fix you up.