This is a bit confusing and annoying, I use castle as my main base, before I took the castle I was able to assign Preston to guard duty.
At the castle it don't work and he is farming all day, same with the other followers.
However I found I could assign Pipper to anything even caravans.
I think this is some messed up flags, Had looked better if you could assigned some of the more combat oriented followers as guards.