4GB+ for hugely modded graphics. 3GB+ for modded graphics with some restraint. 2GB if you're happy with the official high res texture pack and a few other texture mods.
Mods are often released in multiple versions with texture sizes of 1024K, 2048K, and/or 4096K. Many people think 4096 is overkill on a 1080p monitor. If you stick with mostly 1024K textures, 2GB of VRAM might be enough. If you go with 2048K or higher, you'll probably need 3GB of VRAM.
I think you could have a great looking game with 3GB of VRAM, but you'd have to be a little bit careful about installing textures.
The other issue is you have to spend some time to learn about modding if you want to use high res textures. You absolutely must have several helper mods correctly installed, or you'll get CTDs. Bare minimum you need ENB or ENBoost + http://www.nexusmods.com/skyrim/mods/50305/?.
Or you could keep it simple, and go with 2GB of VRAM and the official patch. I'm not trying to scare you off, but when you get into modding this way, you're going beyond what Skyrim was designed for.