It's the atronach perk! I reloaded a ton of saved files to narrow it down. It "absorbs" the attempt at summoning a pet 30% of the time and instead refills your mana bar (which is never what you want at the time your summoning a pet). Just posting this to hopefully help someone avoid the frustrations I went through. If it's already been posted somewhere else I wasn't able to find it myself.