Entering the sub-basemant of Vault 21 when you chase after Benny seems to be the cause of MY problem. I don't remember the door being unlocked the first time I played, so on my second time through it felt kind of odd that it was. Almost immediately after leaving the sub-basemant my game started freezing a few seconds after loading a game, or it'd just load forever.

I fixed this problem by loading a few minutes before I entered and didn't enter this time and the problems have stopped completely. Seems that the saves following my entry and departure from the sub-basemant were all corrupted.

Hope this helps other people.