In FO3 I made megaton my main base and every time I dragged another huge sack of junk through the gates, Sims would just roll his eyes. I scavenged every single thing that was not nailed down in every building in the Wasteland and ended up owning Megaton lock stock and barrel as the wealthiest man in the wastes, in my own mind of course.
They should make it so that once you get a settlement going, you can assign people to tasks like scavenging for materials.
I think the key is to be practical, and have a focus. I am obsessed with the settlement building too, but I also only have a 1 Str. So I have to prioritize. I don't pick up items that will yield common materials like wood or steel. I have a specific "phase" of development, or build plan each time I return to base, and I just collect junk that will be useful for THAT next build.