One downside to no fast travel is less random dragon attacks which means less dragon souls! They seem spawn more if you fast travel. So don't come crying to me when you need those extra souls to defeat Miraak!