Time passes in town as well, but really, really slow. You can leave your character for a while, but it won't be 5 days if you stay away for 1 whole hour.
Time also passes when you travel form place to place on the map. Time will fly by if you just go around and explore. And yes, the PipBoy can also make you wait/rest. And when in a location I do believe it uses real time.