How about rather than just being able to charm anyone into doing anything almost immediately, you had 2 meters. One for how they feel about you right now and one for long term feelings? Charming will work only for short term, and if repeated often enough, for long term, but you have to keep up their trust and keep them liking you as well. And charming should be more skill based because I found it really easy in Oblivion.