Yeah "passive" mode could certainly be a bit more... passive. Had one of my companions run up 2 flights of stairs and around a corner to get at a ghoul...while in passive mode.
Passive prevents them from attacking things that they detect that don't detect them.
It doesn't prevent them from attacking things that they detect that do detect them. That is considered self defense by the game.
The way that sneaking helps is it makes all of you less likely to be detected, therefore your companions are less likely to run off.
Any time you sneak and see "Caution" appear, it's extremely likely your companions are about to run off.