I am on xbox too and the load times are totally improved over oblivion, try installing the game to the hdd and take off the auto save option for travelling.
first of all, what's an hdd? second of all, i don't expect to fix it just for traveling. even if i accidently got into a certain building and i wanna get out it takes a lot of time, but i will try it.