Yeah I have to wonder what the heck I am doing wrong too
Although "carelessly un-installing" mods you would think would be a recipe for disaster, it all depends on what type of mods you have uninstalled, if its just texture / mesh / sound replacers those would be no problem to uninstall, but if any Ref-IDs are now missing or any of the other possible problems http://afkmods.iguanadons.net/index.php?/topic/3676-skyrim-information-baked-into-saves are just waiting to pounce when conditions are right.
You may have papyrus errors and scripts misfiring too which you may not be aware of. There can be a heap of stuff going wrong which do not actually trigger a CTD, events not happening that you may not even be aware of, or long term domino effect of things going wrong, I cant imagine anyone getting away with bad modding practices for too long and remain totally unscathed, logically if things are missing from an ongoing game then all is not well with your setup. But starting a new game would not be a problem.