They used war-hammers the size of small children in Oblivion. And even at a lighter weight like 20 lbs, it's still pretty hard. MY emphasis was on the animations, not the possibility of it being done.
The weights represented in the game have been found out to definitely not be 1/1. I believe the correct translation is division by 4 (which makes excellent sense. Most warhammers in the game weighed 20 or so "units". Dividing by 4 leaves us with 5 pounds. That seems like an accurate representation. You can do the same with the armor or other weapons and come up with very realistic weights.
But regardless of these things, I understand what you mean. Acrobatics definitely should come in to play a lot more than it does. It did come into play in Oblivion, as below 25 skill you can not attack while in the air, although pretty much every character starts with more than than, so it was a kind of moot perk.