1) Is there a way to sabotage the mission by only poisoning the nest and not the brewery?

Sabjorn doesn't say anything else even though I've cleared out the skeaver and poisoned their nest.

2) Is the the only course of action to poison the brewery to advance the quest?

3) Is failing the mission not an actual option in game?

no choice but to do the full objectives, else it don't advance at all

you need to poison the vats so the guard captain arrests the owner. The captain then puts your contact in charge who works for Maven. Hostile company take over. Now if we could have only poisoned the vats and left the nest alone... thus ruining part of Maven's plans.

Then what was the point of BGS putting being able to poison the nest. I like the way the TG was in Oblivion, a little more noble/ class than murdering thugs. Ah well, I guess I'll leave Sabjorn hanging for a bit while I do some other missions.

Err don't you mean poison the nest and leave the vats alone? At least that's the way that I would like to have done it.

well no see you need to poison the vats so the owner gets arrested and the quest progresses. It would be nice to have an option to sabotage Maven's plans a little by not taking care of the skeevers. Then maven just wasted a bunch of resources on a Meadery no one wants to buy a drink from.

If you need an RP justification, then consider this is being done at the behest of Maven who is clearly evil. If you proceed through the Thieves, you may eventually find yourself in a position to affect the guild's ethics.