On the contrary, you might have performance troubles if you had less. Are you on a 32-bit system btw? If so, you must realize that the 4GB RAM limitation includes video RAM. So if you have a 32-bit system and 4GB "normal" RAM, only about 3 to 3.7GB "normal" RAM will be recognized by your system. Skyrim will still be allowed to go up to 2GB by default though, as long as it's available. It's not something to worry about, you'll still reach the recommended specifications, just saying to give you the right info.
4gb is what is supported on 32 bit systems, and xboxs have way less than that and they can still play the game fine
I know little about consoles, but you can't make such a comparison just like that. I run windows 7 64 bit, and right now only have my browser, messenger, a bunch of folders and my standard background applications open and my system uses 1.9GB RAM already. I'm sure the default usage of Xbox is very low. In addition to that, I'm also pretty sure that the Xbox's RAM is used less often, reading more directly from the disk. Running vanilla Oblivion takes about 500 MB RAM on my PC, which is already about the maximum capacity of the 360.