Actually, yes, you can. What do you think banks do by looking at a person's credit rating?

If the person does not pay for a service, they do not suddenly get to use that service when they suddenly need that service. Financially it just won't work. Services need to be maintained in order to function properly - if no one contributed money to maintain those services, those services can't function, and when suddenly someone needs that service, it can't simply appear out of thin air. If someone doesn't help pay to maintain the service, they shouldn't get to use that service when they know they aren't covered by it by not paying and they still refuse / forget / whatever to put money into that service, which, again, needs to be maintained.

Surely then, we'd see more foreigners getting mugged in the streets and the police just watching? I mean, I don't recall foreigners paying any sort of taxes. Yeah, just let them get mugged, bunch of free loaders.

Let me break it down:

1) Someone is breaking the law by mugging someone else. Foreigner or not. Visitors to the United States are subject to its laws, as is everyone else.
2) The police are paid to uphold the law in their jurisdiction.
3) The police stop the person from breaking the law.

Yay, Post limit.
Don't start another.