After you turn over the evidence, both the Van Graffs and Alice are taken away. There is no one for Cassidy to go after. However, the guards at the Silver Rush will go hostile and attack you the next time your on the strip. So, that problem takes care of itself and no Karma loss. Alice is not at the Caravan anymore, so nothing to do there.
As for Cassidy stating her desire to take care of business, ignore it or take her to Freeside the next time you go so she can take out the Van Graff Guards.