Whenever I try to look left or right with my controller the crosshair will catch on walls and slow down immensely. If I press the joystick again it'll resume at normal speed. I've tried this with a vanilla game and it still persists. With KB and mouse I do not have this issue.

While I can't recognize your specific problem I agree that controller on PC feel very weird and wonky.

With Windows 10, the new XBox One wireless controller and adapter from Microsoft work wonderfully. No lag, jitters, or anything else.