So instead of activating a ladder and being instantly loaded into another cell, i'd have to activate a ladder and waste time climbing up and down? Tedious. Waste of time. No thank you.
I can agree here while the ladder leads to doors that are transition to another cell. There's no point of wasting the time on climbing that ladder, for sure.
BUT... if the ladder leads to a place that is exactly in the same cell then having any loading screen or teleporting system will be an immersion braking.
Just imagine you wander through the mountains and then you spot something flashing up there high on vertical slope.
And it will be a cosy camp with fireplace and a nice view on surroundings - all this in a small rocky cave, a carved niche rather than a proper dungeon.
And the place is only accessible by climbing a ladder which was left there by a previous camp owner.
Of course, ladders are by no any means a crucial game's element (at least not for Skyrim), but they can just add that "third dimension" to the world
which allows you to move vertically and access or discover many thing up there which aren't normally seen from the low perspective ;]