Yes, you need to be able to work on enchanted items, and then you just need the leather to be able to do it. I improved both, but having no perks in light armor it was pretty pointless.
You need the perk that is straight up from the first perk in the smithing tree, believe it's called Arcane Blacksmith or something, that allows you to upgrade enchanted weapons/armors