I found that i have to constantly bring along soul gems to recharge them since they deplete rather quickly. Is this supposed to happen or is this a bug?

It's normal. It's quite bothersome