I want to make a feature in my mod so that skills you do not use for along time start to get "rusty".

Not using a skill would slowly decrease a skill such as Block. The mod would NOT be responsible for increasing the skill again.

That would be done by the player using the vanilla mechanics such as training or just plane using the skill normally again.

So If I want to reduce the skill in a way that it could still be increased via normal game play would I use SetActorValue?

I am also do not want this to show in the stats as a "damaged" actor value in the menus.

Like this:

And why not check both and see what's changed? Mod will raise the value past the base(or maximum value). You're probably better off with Set/Restore, or even use DamageActorValue. I used that in my HP convert tests, and got no red numbers.

Reading your code, if One handed was 50 points, there would be a 5 point increase if my math is right. Buffs could be a problem. I don't know how to calculate around those. SetActorValue will keep them intact too.