I'm a little confused by the concept of escape velocity so I was hoping someone could explain. For example, I understand that if you throw a ball up in the air, it will eventually come back down due to the pull of gravity. It would have to be thrown at Earth's escape velocity in order for it to continue onwards.

But my confusion starts here. Suppose, theoretically of course, that we could build a ladder 100 million miles high. Once we were at the top, we jumped off. Wouldn't we have escaped? Or is it that the gravity of earth is infite and even though we are now 100 million miles from the Earth, we are still under the same gravitional field?