Well, it depends.
By nature, the game doesn't force you to do anything, but if say you decide to join the Mages Guild, there are parts in which you absolutely have to cast spells. You don't need to specialize in magic abilities, since scrolls are there, but the fact is; yes there will be points in which you need to cast magic, but only if you throw yourself into such a situation. If you stick to things like Arena and Fighters Guild, you will never have to cast magic.
They basically left a lot of stuff open. Anyone can do anything, that's what's so brilliant about the game. Even mages can become successful dungeon divers. If they have a high destruction skill, strong feather spell and good shield spell; they are just as effective as heavily armed fighters, especially if they have high conjuration, they can even summon an entire daedric suit of armor and a daedric weapon.
Similarly, a warhammer toting brute can become a stealthy little rascal if he simply sips on an Invisibility potion.
And stealthy sneaks can perform anything a mage can, simply by buying (or even stealing) a scroll.
See, it works out well no matter what.