I've done "Defend the Castle" 3 times now after informing the BoS about the Institute wanting the beryllium agitator, and even if I attack BoS before starting that quest, I only get the Institute attacking.

I've read a quest guide for the game that says that quest has Super Mutants, BoS, and the Institute attacking in waves. Other guides, like IGN's, says it's only the Institute attacking.

If anyone's had a scenario with 3 different factions attacking in waves in one battle, how did you set it up?

Ghouls, super mutants, raiders, and gunners attacking the Castle are settlement type attacks like "Help defend Abernathy farm". You can't control when you get them but if you do get a settlement defense quest, there's a chance it's the castle.

So far, ghouls and raiders have attacked the castle. The ghoul invasion was epic destruction.

The BOS only invade if you do "With our power combined"

The Institute only invades if you do "Defend the Castle" for the MM questline

The Defend the Castle quest I'm talking about is not a radiant one like you suggested, it's a Minutemen story quest that is only triggered via doing prior story quests a specific way. I'm sure the guide I read was wrong about it though, because I don't think it's possible, or even sensible, that 3 different factions, especially ones that don't coordinate with one another, can or would attack one spot all at once.