I'd find it hard to go through the game completely hidden, especially at the start. In interiors it's manageable, but when you're in the wilderness and get ambushed you'd find it extremely difficult to survive without any armour.
Give it a go though, see how it turns out. You could perhaps wear light armour in unknown territories, and then clothes in interiors and everywhere else? Just a suggestion.