I play on my Xbox 360 so the wait times aren't as miserable. I'm assuming you guys are playing the PC version. But the if they do get slow then I'll (ashamed to admit this) start singing along with the background music.
I play on the PC, and my load times are comparable to those on the 360 with about 120 mods installed and roughly 20Gb worth of asset data. Even then, the only really long load times are on the initial game load and the initial exterior worldspace load. And those only take about 20 to 30 seconds. With 120 mods, some of which are HUGE. At this point, I would say on average, my load times are about ten to twelve seconds.
Completely vanilla, the load times are only a couple seconds. Ten seconds at most.
If you have a good PC, it will MURDER the 360 in terms of performance and load breaks.