I mean a pure mage using NO weapons except for his hands. Is this a viable build or would I not make it out of Seyda neen?

Totally viable. You can probably get your H2H skill up to 60 or even higher just by beating mudcrabs around Seyda Neen. In fact, that's what I would recommend. At that level you should be able to take on bandit types pretty easily.

I have to ask: Do you consider Conjured weapons (i.e. bound dagger) to be spells or weapons?

I am experiencing h2h for the first time and its pretty damn op. What spell skills are u planning on using?
Are u wanting to be a melee fighter that uses spells for def and misc stuff or a mage that relys on his fists when magic is depleted?

No I wont use conjured weapons.. I want to roleplay a monk that can control the elements, kinda like Ryu from streetfighter so mainly destruction.

Hand-to-hand is strong against most NPCs. If you can sneak up on them, you can get a critical hit, which often knocks them out. High agility helps reduce the number of hits you take before you knock them out. Bloodmoon adds the possibility of becoming a werewolf, which gives you much stronger h2h attacks while in werewolf shape. There are also mods that can enhance the experience.