I play several non-armored characters in all the Elder Scrolls games. Once I get a good feel for playing a game (I don't try this on a first character), it's one of my favorite things to do. I've played no-armor thieves, mages and even warriors who use no magic.
First and most essential, in my opinion: play smart. Your first order of business should be avoiding getting hit. Be cautious. Think ahead. Sneak or use Invisibility (if these are appropriate to the character). Use terrain to your advantage. Pick off your enemies one at a time if you can. The one thing you do not want to do is run head-first into danger.
You can do this with any combination of magic or any combination of physical weapons. You don't need a shield. I've never used a shield in any of these games. Use one if it feels right for your character. I've done this with characters who wielded claymores, or bows, or pure mages who used only spells.
The hardest of these characters was a claymore-wielding Barbarian character who feared magic and refused to use it. That meant not only no spells but also no potions. I played that character for about 150 hours and it was great fun. But it kept me on my toes.
I would think a Spellsword would be fairly easy since you have a great many tools at your disposal to stay alive. Good luck!