Yeah so Ive done more digging and im pretty sure its the infamous memory leak issue. I swear I remember this being a huge issue with skyrim too especially when I started downloading a bunch of mods. You would think they finally found a way to control it better by now. Not to sound like a [censored] but I NEVER had these issues with witcher 3. It started out pretty good, then got really really good after that first big patch. But its early, so time will tell I guess...