There is a repeatable bug related to the idiot savant perk. Any time there is a delay between the perk activation (noted by a sound effect and pip boy graphic) and the receipt of the experience (noted by the experience bar display counting up) the bonus is not applied. For example completing a quest that causes a notice that your follower liked that prevents the experience point display to activate for a few seconds until the affinity notice fades.

My guess is the idiot savant perk is actually a 3 or 5x boost for 5 seconds when it activates, which is usually just long enough to affect the quest completion, but in about 5 scenarios I've experienced it activated but had no effect on the amount of experience gained.