Nope.
I thought, at first, that AGI is what affected your speed and most Medium/Heavy armor have -AGI stats on them; but you go slower in the T-51b Power Armor, too, which has no ADI penalties. It's just based on the armor class (light, medium, heavy.) If you want to move faster, wear lighter armor.