They take an incredibly long and convoluted way to the memorial. It's not a glitch. If you decide sometime, you can escort the caravan to the memorial. Be warned though! I had to constantly run in an oval racetrack pattern around the entire caravan. Because of the presence of the player, enemies spawn all day long while you are doing an escort mission. The best I've ever done is only having two ex-slaves die. I didn't reload that play, as the annoying girl who challenges you at the gate was one of the dead.