A couple of things about graphics components in mods.
Whether PC or Console, your equipment is only capable of displaying certain resolutions. These resolutions are limited by the graphics components in your box (PC or Console) and the display you are using. If you exceed those limits the textures will be interpolated to a usable resolution. This requires processing time by the hardware and often results in a picture quality lower than what you would have had if you had used textures more appropriate to your hardware. In short using too high of a resolution texture can make the output look blotchy.
On the PC, the graphics components reside in Video RAM when they are in use. There are also graphics components from the base game in VRAM. If Fallout 4 requires 1.5 GB of VRAM to run, and your PC has a 2GB video card, that means you would only have a half gigabyte of VRAM available for graphics assets from mods. If you exceed that, the GPU will go into a thrashing state where it is spending more time moving assets in and out of VRAM than it is spending on rendering frames. When that happens, you will be lucky to get a framerate of eight frames per second.
When it comes to Consoles, I am not sure how the 8GB of memory is allocated or addressed. I do not know how much is used for various system tasks, how much is available for games and out of that how much of that can be used for VRAM. Ultimately, it doesn't really matter. You need to understand that everything going on in the console is going to require some of that memory. Even internet communications is going to eat into that 8GB. In many cases it won't be much, but it all adds up. Fallout 4 is going to occupy a large chunk of memory just for program code and variables. What is left might be usable for graphics assets. You will not be able to exceed the 8GB limit. And every mod you add will eat into that 8GB limit (either as graphic assets, data assets or both).
I am unfamiliar with consoles, but they are essentially computers so all the the general rules that apply to PC's also apply to consoles. If Fallout 4 with mods needs more RAM/VRAM than is available something bad will happen just like on a PC. I don't know how an XBox One would handle it, whether it will start swapping to disc (if this happens expect your framerate to be measured in seconds per frame and not frames per second), you get dumped out of the game or if the system will lock up and you will have to restart. But you WILL be limited in how many mods you can use and what type of mods, just like a PC. The limits will probably be different, maybe even better in some cases, but they WILL be there.
As for script extenders, that is something that is in Microsoft's court for XBox One.