It is too bad you can only hire the total of 5 guards (a captain and 4 men at arms) and that you can only have one follow you outside the keep and he can get killed very quickly. It would be great if he was only knocked out and get stronger as you level up. If you lose one of your men at arms in a fight, do they get replaced?
Also if you ever wanted to upgrade this add on Fighters stronghold. You could make it so there were villages and inns that were under your control around the castle. They could be along the road that the castle is and the road could extend into the next territory.
The villagers could work for you and pay taxis and you could protect them and their could be missions to protect them and the like. Maybe even upgrade the villages themselves as well as hire more troops to patrol. Or even add more villages to your domain.
You could even carry this idea over to Skyrim and try it their. What do you think?