I'm using my 3 and 4 mouse buttons without issue. I have them mapped as Generic Button in my mouse utility. I've used them in Fallout3 this way, and also mapped to a key. When I had them mapped to a key, I had to map
that key to the function I wanted--the game ceased to recognize the click as a mouse button, but registered it as the keystroke that it was mapped to.
So, you could try mapping button 3 to V in your mouse utility and leave Fallout with V as the VATS button (in the keymapping section), or you could set the mouse button as a generic button and map it to VATS in the mousemapping section.
I assume you know that you have to click Device in the bottom right corner of the menu to switch between key mapping and mouse mapping.