I agree, for the most part. But be honest- given most things being this well-hidden, the "rare and occasional" http://www.uesp.net/wiki/File:BM-interior-Fjell.jpg does have its own fun factor.
True, I wasn't saying for all caves, and even ones with good hidden passages could have some kind of marker easily missed like your example. I remember my friend trying to find a riddle on Batman Arkham Asylum. I saw it right away because I gathered what the riddle meant. But I wouldn't tell him where to go because it was fun seeing him sift through the clues as he wandered through the enviroment, piecing it together.
Btw, it was a Riddler question mark that has the main part of the question mark on an doorway and the dot on the ceiling of the room that it led into. You had to line it up just right to get it to work. It took my friend just under an hour to find it lol.