Recently I been back to playing skyrim, and I recently realized it takes like 5 minute's just to load my game. Is this suppose to happen? Is there anyway to fix it?

If you have a lot of save files it will cause slow loading. Try deleting (or moving to another directory) all but 20 saves. Also how big are your save files, if they are over 15MB it might be the reason.

Well I only have one save file and it still takes 5 minute's to load, even if I travel across the map.

Is the save game excessively large?