I actually like the rocks. I try and only get rid of them if I need to, otherwise I'm falling victim to the irrational sense of order which caused the victimization of digital entities in Tron.
I also have a suggestion for Bethesda that rocks should reappear in areas that haven't been built into, and that there should be a rock match ing mini game with different types of rock and ores that have a chance of appearing.