http://www.uesp.net/wiki/Skyrim:Fortify_Illusion
Talk about immersion breaking...
This is exactly the type of situation where I wish there was an in-game solution. Don't get me wrong, I've used the UESP wiki many, many, many, (many) times and it is essential for uncovering so much of the game. It would just be nice if a tavern worker happens to mention a rumor about a hag in a cave who knows the secrets of illusion magic...
You can already pick up rumors from the NPCs, I just wish there were more of them. Fingers crossed for ES VI.