Maybe if I could craft specific keys, and make key sets. Then I could pass them out to only those npcs I want to have access.
I'd also like to designate 'personal areas' where npcs won't go. It drives me up the wall to walk into 'my' house / room and find people sitting in my space or sleeping in my bed unsolicited. If however they're my current follower and/or lover then that's fine. Mi casa, su casa.