I like changing between 3rd and first person often with the toggle POV key as it's faster then scrolling out with the mouse, but it seems to only work when the camera isn't moving at all. If I'm moving the camera even slightly with my mouse when pressing the toggle button, it doesn't work. I've tried changing the bind to a different key but it doesn't help. Anyone else have this problem?

Yes, it's just another thing bethesda needs to fix.