My esp alone is pushing 10 megs, when should I start to worry?

I would only worry if it would noticeably slow down the game somehow.

You should only worry if the size of your plugin seems inappropriate for what's actually in the plugin.

As a benchmark, most detailed house mods clock in around 200 kb, castles can be twice that. UL mods might be anywhere from 700 kb to 2-3 MB. Kvatch Rebuilt's plugin is 824 kb, which is actually pretty trim for a city, as most village most seems to hover around there. A particularly complex companion might be 2-3 MB, while a very simple one might be 200 kb. Scripts will make your plugin bigger faster than anything else, in my experience, with new records being a distant second.

If your mod doesn't contain a lot of scripts or new records, you might want to have a look at it in TES4Edit to be sure you've not got data in there that doesn't belong, such as duplicated cells or re-saved scripts (the dreaded "Compile All" button can easily put your mod at 10 MB). If it's just a big mod in general, however, then don't worry at all. Oblivion.esm is 264 MB, and the game runs just fine with that loaded.

Well, its huge. Perhaps 100 NPCs, 10-15 worldspaces, one of them about 1/4 the size of SI, many new quests, probably over 100 new scripts.

Thanks again.