Forget the fact you are the Dragonborn, you are an adventurer in a low technology society, with a whole school of magic devoted to ( the clue is in the name ) restoration. How does a mechanic that makes you exhausted just from standing around for eight hours make any sense?
And restoration can restore stamina. But, just standing around
does get tired.
I'm not even saying you have to "Sleep" to get stamina back. I define resting as Oblivon defined "Waiting". It can be done anywhere, by either leaning against a dungeon wall, or sitting down wherever's convenient. But, it shouldn't be done passively in combat, unlike in Oblivion or Morrowind. (That's not to say you shouldn't be able to get a brief respite when being pursued by going into a non-menu "Rest mode" such as sitting)
On that note, there should be a simple button to let you sit down/recline anywhere to recover stamina in real time without using a "Wait X Hours" menu.
There should be consequences for low fatigue (Unlike in Daggerfall), but it should take a
lot more effort to go into Low-Fatigue mode, with the Fatigue/Performance ratio being inverse-exponential instead of linear so there's hardly any difference until you start over-exerting yourself. And when you go down at 0 Stamina, it should force you into the aforementioned "Real-time Rest mode", but not get out of it until you recover most of your stamina again. (Due to the quirky nature of potion consumption in The Elder Scrolls, you can still drink Restore Fatigue potions when unconscious in this manner)