What level are you? What stage of Vampirism are you at? Stage 4 Vampire Lord is significantly stronger than stage 1.
Keep in mind the Vampire Lord functions similar to Wizards and mages. Their main source of damage comes from ranged magic attacks. If you are using him only in melee combat, you are going to have problems. The Vampire Lord is extremely agile and mobile, use the hovering to your advantage.
This may also be the problem you are having. There is a glitch where if Serana is your active companion, your Drain orbs will do significanlty less damage. The only remedy for this is mods. Or don;t have her as your companion... which may not be possible for the majority of Dawnguard's MQ.