Invisibility was a master level spell with a hefty skill requirement.
You didnt have access to it fresh out the sewer, you have to purpously work to attain the spell and appropriate level.
I have been level 40's who couldnt cast invisibility.
You can choose not to train your illusion.
I have no sympathy for people who clamour to rid the world of something they dont like but lack the self-control to avoid,
thereby denying people that do like it the content.
What you said.
Honestly, for all the noise about immersion and experience breaking etc. consider how immersion breaking (and potentially frustrating) it would be if you spent vast expanses of time making your character into a master mage or a master of illusion magic. . . and then couldn't do anything with it worth writing home about? If you were no more special and had no more advantages over the Average Joe NPC than you did before you mastered them? THAT is immersion breaking. It is immersion breaking to have a game set in a world where it is known that certain, powerful forms of magic have the ability, at very high levels, to turn people invisible, and then have a character who is a master in that field of magic unable to cast such a spell.
It would also be immersion breaking if every single hostile NPC in a necromancers robe, or who has ever so much as stepped into a mages guild to use the bathroom, is automatically equipped with a powerful spell for detecting rarely encountered invisible beings.
Consider. Let us say someone goes to see some Harry Potter prequel movie, and in it Albus Dumbledore himself is tasked with recovering something from a stronghold of dark wizards etc. etc. etc. Now anyone who follows that series enough to bother seeing such a movie would come in with an understanding of Dumbledore as the greatest thing to happen to wizardry since Gandalf and Merlin, a genius, a prodigy etc. Rumour has it he doesn't need invisibility cloaks (rare finds in their own right) to become invisible, because he can cast a perfect invisibilty spell, etc. Now, for the sake of keeping things interesting, it will be accepted and expected that certain rare and terrible creatures (say if a Balrog found its way into Potterdom lol), and the most powerful evil witches/warlocks, might have ways of bypassing his invisibility spell. But if the moment he shows up every wicked witch at the fortress casts some spell that essentially negates his invisibility charm and he ISN"T able to override their detection. . . everyone in the audience is going to put on their WTF face, and say, "what the hell? I thought he was the most awesome, badass wizard on the planet, all of a sudden he can't slip past a couple of low level goons who aren't even supposed to be on the same playing field with him?"
A similiar dynamic applies here. Masters of fields of magic are not common things within the game, and there are many good reasons for that. One is so that the gameworld is not overpopulated with super-wizards. Another is to ensure that mastery over these fields is an uncommon and thus special thing, and an extension of that is ensuring that the player FEELS special when he or she does attain the level of mastery. If every other mage in the game becomes a mastery of magic the instant the player does, it robs the player of a sense of accomplishment. Likewise, if a mage oriented character attains mastery of certian magical fields and then is not able to show himself/herself to be something more impressive than both the average NPC AND the average NPC Mage, then their achievement is muted. Part of the game experience is having the transition from minor to mighty, which cannot happen if the game automatically negates your advancements by enforcing an insurmountably level playing field all around you.