I have a rather strange problem where my frame rate will simply get worse as I play the game, to the point where it is at least 20 fps lower; at that point I can just restart the game and it is reset. Could it be related to memory or scripts? I haven't experienced this on F3 or Skyrim.

While it might be scripts, I'd tend to suspect memory since it's an ongoing process. You didn't say if you're on PC or console, but if PC you try entering the console command PCB (purge cell buffers) and see if the issue clears up. Or it may be related to the 2 gig memory thing, and getting the 4 gig mod might fix it. If you're playing console, I have no idea what to tell you...