Generally castles are built to keep people out, restricting entry to just one gate where they can close off when being sieged.
Did you ever try to steal stuff from a castle in Oblivion?
Sure they might be designed that way, but secret exits were desirable for escape from an unsucessful defense or to smuggle food or rations in for continuing a siege.
Besides, sometimes the walls get worn to the point where thieves could climb over some walls.