Settlements are a massive mixed bag. There are a million things that I would change to make them a lot more interesting and dynamic. But more on that in later posts.
For now, I just have to say that settlement management is a pain in the ask. I would venture to suggest there should be some kind of operations desk that has an overview of settlers, their armament, their task, happiness and psychological state in list format.
Prison Architect handles this extremely well. It would be a set of extremely basic intelligence and management tools that would dramatically reduce the time invested in organizing settlers. Because, as is, it just feels like superfluous busy work.
I would also suggest that it be one of the benefits of the Local Leader perk or a settlement addition added after your first few farm conversions.
Just a thought. Argue amongst yourselves.