After the Dawnguard release and playing it a bit with my Necromancer character I remembered that Vampires and Necromancers dont actually get along quite well. Oblivions faction system had them hostile to each other. I also dont think most vampires would be happy about a Necromancer coming to their place trying to control them and many Necromancers would see vampirism as a failed and imperfect state of being because you still have weaknesses and a thirst for blood.
So is there actually something in Lore which states their relationship with each other or is it just mostly gameplay mechanics which make them look like rivals?

No. Its just the mechanics

Lots of Vampires use Necromancy though.

There is actually a thread about this http://www.gamesas.com/topic/1248589-why-do-vampires-and-necromancers-hate-each-other/page__p__19251581__hl__necromancers__fromsearch__1#entry19251581.

I would say their relationship is "strained" due to the fact that Necromancers see themselves as the (living) lords of all undead, while the Vampires are 1. Undead and 2. See themselves also as the (undead) lords of the undead.

Thus there is friction and fighting between the two groups.

I'd like to point out, from a very similar thread in Skyrim General, that....