Is it worth leveling my heavy armor skill to do? Ive already put 3 perk points into Light Armor before I found out about this.
I'd say so, yeah. Unarmed characters that take advantage of every perk that helps them (Fist of Steel, Well Fitted, Matching Set) hit pretty hard, and fast, too. You can still wear light armor with heavy gauntlets, if you want, but you'll never be as tough as if you were wearing heavy armor, head-to-toe. If you want to wear light armor, I'd suggest decking yourself out in heavy armor once in a while and letting yourself get hit my a mudcrab over and over to bioost your skill. Takes a while to do naturally wearing only heavy gauntlets.