There are several things to consider before deciding to wall off a settlement:
1. Each settlement has very specific spawn points for enemies.
Walls can be extremely effect if you make sure they are between your settlement and the spawn points. Sanctuary Hills has a spawn point in front of the bridge out of town, and one behind one of the houses. Since Sanctuary is so big, it is easy to accidentally include enemy spawn points inside your walls.
The Slog has two enemy spawn points - one at the crossroads just outside of settlement, and one on the hill behind the swimming pool / tarberry pit.
Knowing where the specific spawn points are before you build your walls will save you time and grief. I personally recommend letting your settlement get attacked twice before building walls, just to pinpoint where enemies will come from during attacks. In my experience, this will stay the same for that particular settlement.
2. Build Size versus Benefit
If you are playing on console or a weaker PC, Build Size is a concern, and you have to decide whether or not it is worth it to have walls around your settlement or more buildings and decorations.
If you know the spawn points, you can always just plop down some turrets near each spot, and as long as you are in-town, any attacks will be gunned down before they get their act together.
3. Realism
I like to build walls around my settlements because it just FEELS right. Like someone else said, if you lived in an area with Super Mutants, Radscorpions, Vicious Mongrels, Mirelurks, Deathclaws, etc. I'd sure want to live in a walled community.
I also use the walls to create choke-points. I always put my gates close to the enemy spawn points, and then realistically man the gates with defensive structures, guards, and turrets.