So I just reached level 11 and have people at 2 of the settlements (Sanctuary and AF). I stopped the people at Sanctuary at 7 just because I wanted to wait on getting a ton of people there, and just have the 3 at AF. I've only had the mini quest at AF to defend it but not had any other raids. Does it happen more often at higher number of people or level? I do know about the defense mechanic for lowering raids. So was wondering if there is a trigger number of settlers or something before attacks can even happen?

Other than having your defenses lower than resources, the attacks seem to be random.

I've had hordes of ghouls attack a settlement with only 2 people... and I've had raiders attempt to ransack Starlight with 14 people and 10 turrets. I've had super mutants in Sanctuary.

Some have happened when I'm present, some when I've fast traveled, and others while I've just been out exploring.

Totally random it seems. Some people complain they have them all the time, others that they've never seen one. I will tell you that I've had more on my second playthrough than in the entirety of my first.