That and I just hate coming out of a dungeon and having no idea where my horse is. "Now.... let's see.... I left it in some grass next to a tree near some rocks-- that should be easy to find, right?"
I just set a compass marker on the spot where I've left my horse.
A couple of other pros I haven't seen mentioned:
1) You can climb almost any slope no matter how steep when on horseback, makes traversing mountainous terrain much easier.
2) You can carry huge hauls of loot over long distances much more easily on horseback; once you've mounted your horse, it doesn't matter if you're over-encumbered. So, rather than having to quaff a whole bunch of feather potions to get you home from some ruin in the back of beyond, you only need enough potions to get you to your horse and it can take the load from there.
I tend to use horses more often than not, enjoying the extra challenge that comes from trying to keep them alive. Also, there are some neat things you can do with on-touch spells to boost their usefulness, particularly using a weak fortify speed spell (no more than 5-10 points is necessasry, and even 10 is probably overkill) to make even paint horses as fast as black horses, and (even more so) using water-walking; there's something inherently cool about riding a horse down the Niben (add 3 feet of Light on self at the same duration as the water-walking to give you an indicator of when the spell needs to be re-cast, as horses are lousy swimmers).
I do wish they had saddlebags though, would make them much more useful.