Nah, I'm not convinced this would make the game more fun. You might as well have cold weather hurt you when you step outside without proper attire. It's realistic, sure, but also just another annoyance to solve with [insert magic buff here].
Wouldn't really mind that either tbh, I realize some would find it an annoyance, but to me stuff like that makes the game more fun