So, there is a problem with follower commands and controls. If you have your control mapping set to have activate on the "B" button just like me, then when you tell a follower to do something they just go back to normal.

Example, I told Lydia to do something and the cursor came up that was commanding a follower as normal. I Told Lydia to pickup a piece of moonstone ore (she definitely wasn't over encumbered) and she didn't do anything. She said "Is that it" and went back to normal.

P.S. It doesn't work by holding the activate button and telling her to pick it up either, Also, she doesn't do anything like like attacking or sitting or looting etc.