You can get destroyed at lower levels OR higher levels. It depends more on your playstyle and combat choices than the armor you are wearing.
Smith that [censored], take relevant perks, and play smart. You'll be fine.
unless, you are a smith merchant then you only smith your gear once you qualify for it through appropriate perking.
That's how I usually play and I get by fine. I have an Orc berserker who wears the shirtless fur armor, hide bracers and leather boots who dual wields 2 Molag maces and he handles well with proper perks....it took a long time for his smithing to get high enough to upgrade the maces as I only smith here and there with animal hides I collect on my journey.
I once did a fun, but short mind you, restoration only playthrough. I think you can get by just fine mate.