After trying pretty much everything to eliminate the crashing to desktop, I decided to try bumping my CPU voltage and uncore voltage, even though im running my 980x at stock speeds. To my surprise this has completely cured the crashing and I was able to add back a hefty GPU over clock with absolutely no issues whatsoever.

Sadly for me, the low fps spikes in certain areas remains, along with the patchy GPU usage.