The worst thing is that your character becomes unplayable due to CTDs and bizarre bugs.
The general rule is: don't remove mods from a current character and expect to have a playable game. This is especially true for mods with scripts. Skyrim stores references to those scripts permanently in your saves, and will continue to try to access those scripts after they've been removed.
You might be able to continue with that character for 20 or 30 more hours, but they will eventually become unplayable. That's from both personal experience, and a lot of posts over the last couple of years.
I haven't used it (yet), but http://www.nexusmods.com/skyrim/mods/1334/? seems to be the favorite for people who want to have multiple characters with different sets of mods. Wrye Bash is great for mod management, but probably a little harder to use and maybe not as good for different installations used at the same time (though it does support this).
To manually remove a mod, you should open up the original package (with 7zip), find every file it adds, and delete each corresponding one from the Skyrim Data directory. This is very important for any scripts it adds, especially the ones that override vanilla scripts.
You have leeway with certain types of files like textures or meshes, but scripts and animations at least should be carefully removed.
Another option is to either reinstall Skyrim and reinstall your mods with the Mod Organizer or Wrye Bash, or to delete every directory under the Data Directory, and reinstall all your mods, again using either MO or WB. I say this because MO & WB will track every file's relation to every mod, so you get 1-click installs and uninstalls, and even options to configure which mods override others, when there are file conflicts.