Well, it's not like they all trust you. A lot of quests I have gotten have just been me over-hearing a convo, or a guard making a passing comment. Suddenly "Map Marker added". Or from reading a book. I'm okay with it, although it does clog up my quest log sometimes.
I really dislike that the Miscellaneous quests have absolutely NO useful info attached to them. So let's say you did a quest, however you happened to get it, for a guy named Shadr. You might get distracted right after completing whatever the quest task was (for example, running off to chase butterflies... not that I would ever do that, noooo.) And a couple hours later, half way across the map after doing a dozen other things, your Miscellaneous quest log only shows "Talk to Shadr". You have no clue whether talking to Shadr will give you a quest activity, or if it is part of a quest you were doing and talking to him will progress things, or if you finished something for him and that will end the quest. And there is no info on what exactly you did, or where Shadr might be located. You have to select the quest to track, then look for a map marker to even have a clue as to his location. So, yeah, frustrating to me. I try and keep things manageable on my current character and avoided Riften for quite a while, since that place seems to be a den of minor quests

I wish there was a way to abandon some quests, when they get forced into your log.