Agility also affects your running speed, which is half the reason I put any points in it at all, since I don't use VATS.
From what i've tested so far, there's absolutely no difference in running speed between a 1agi character and a 10 agi charater.
I made both, ran between 2 telephone poles and timed it. 20 times and no difference in time.
Agillity does however effect weapon drawing and reloading speed.
Personally i play a sniper kinda character. I've got 2 Agi since i need to have 1 and get 1 from the implant. Never use VATS either and the only 2 perks you will miss of any importance, are Silent Running (which when you think about it, isn't that important anyway, since you'll probably be at a good distance from your targets) and Fast Reloader, which isn't really needed anyway; most "close" combat ranged weapons have several rounds in a mag and the only 1 shot sniper rifle i've found so far, is the Anti-Materiel rifle, which can deal enough damage to 1 or 2 shot everything. The regular sniper riffles have a 5 round mag so they don't suffer either.
Shotguns tend to be bad with low reload speed, but if you intend on being the long range rifle guy instead of the close gun 'n run guy, you won't be needing those anyway. Not unless you're a bad aimer.