It's possible that the ants didn't have the right DNA to mutate, as it's at least shown through all the games, that DNA heavily influences whether or not something mutates. For like every one ghoul that's 'reborn', dozens of humans die from massive radiation exposure. So it's probable that this applies here.

It's also possible that the ants were wiped out too. A bunch of gen 1's wouldn't have much of an issue wiping out an ant colony. Plus, all it takes to kill a colony, is wiping out it's queen, and the eggs for safe measure.

It's a long standing mystery where the giant ants went. In fact, the good people of the Commonwealth sent all their robo-brains off on a quest to find them.