Lack of System RAM is going to cause this problem to happen more often. Again, how many icons?... rows is meaningless because you can change the size of icons and desktop resolution factors. If its more than 30 you should really condense them. I would use folders instead of some of the icons.
It isn't ALWAYS a problem because the System RAM may not be completely used up when switching back to the desktop. The delay is because Windows needs to wait until the RAM is usable or use the much slower page file. Also the size of the icon cache may factor.
***
Without buying more RAM you could perform a clean boot. Basically you need to remove any extra services that you don't need. Your AV and/or firewall will take up RAM so they could be part of the problem. I don't recommend that you uninstall your AV/Firewall though.
How many more seconds or minutes is it taking? If its only a few more seconds I really wouldn't worry about it.