I have been working for quite some time on a plugin that works off my master file. At first it was easy to update with an esp but after a while I noticed the esm references I deleted would remain and the new ones I added/edited would still be there so I would have for example an npc where I first put him in the master. I later deleted him and placed a new one in his place. It looks fine in the cs but when you go into the game they are both there. I really need help please! Basically I cant delete the esm references?

I give you a hint: saved game = a mod always loaded last

Uhh, I don't get it? Is the saved game messing it up? Ess files are loaded last but I still don't see what I should do. Basically I'm lost.

Would converting either file help?

Try your changes from a fresh new game (no .ess loaded last), and see if it works like expected, it should.
If you need your changes to work with previously saved games, try disabling/moving old NPC references from scripts, not from the .esp (which is overridden by .ess changes)
sequence would be
.esm places NPCs
.esp deletes/moves/places new NPCs
old .ess (if loaded, overrides .esp usually causing doubling)
scripts (executed after .ess loaded, can change things)

Abot is correct, it's always best to test mods with a New Game otherwise your game displays the new changes and the references your ess file has made... which is bad.

You can also Update your Save file in Wrye Mash, then use the Repair function also. This updates all the references in your ess, and will clear out any orphan content.
(This is handy if you need to use a certain high-level character)

I actually found a rather elegant solution. I created a mod with my master file active deleting some refs. I then used tesame to convert it to a esd file. I then fused it directly to the esm by adding it in tesame and saving it as a master. I dont know if my question was phrased badly but I meant I could delete anything from the esm while creating my plugin. What is strange is it was that one specific ref I couldn't delete.

EDIT: Oh and I would of deleted it with scripts but it wasn't unique. Thank you very much for answering my question though and I really enjoy your mods abot! Thank you too Pluto because I do use a high level save so I probably should dl wrye mash.

No one should use mods without Wrye Mash... it has about 4-5 essential tools that you need for playing a healthy modded game.