I've played New Vegas quite lot with low Agility and hardly any V.A.T.S. use. It frees up quite a few perks and S.P.E.C.I.A.L. points, I'd have otherwise used for V.A.T.S.(e.g. Commando, Sniper, Action Girl,...).
Has anybody ever played FO 3 without V.A.T.S.? Is it considerably harder? Is it less fun?