It is best to add all the mods you'll want to play before you start.
Removing a mod when you're done playing it is
not recommended unless you know what you're doing. Not only will you lose all the items from the mod, you're going to get problems with missing references.
If you do need/want to remove a mod mid-game, be sure to update and repair your Save with Wrye Mash. (and recreate your Leveled Lists)
Having the MW Code Patch installed will also help prevent problems when adding/removing mods, but it's still a good idea to use Mash to be certain your Save is updated.
And if you want to remove a mod but keep the items you gained, you'll have to make a new ESP that only contains the items you want, then deactivate the original ESP.
Adding mods mid-game is simpler. Just be sure to recreate Merged Objects and Leveled Lists just in case. (and update your Save with Mash)
Removing or adding mesh/textures replacers is almost always going to be safe, since they hardly ever alter or create any references.