I miss it almost every time I use magic in some way.
Every time I finish with the Alchemy and Enchanting tables.
Every time my destruction spells have not done the job,
Every time I my use of magic results in one or more enemies by my side pounding away on me with melee weapons.
Every time I'm in a middle of a fight and I think how a spell could of been better suited to my current situation if I could just adjust it.
Every time I wish I could turn my companion invisible.
Every time I have no choice but to abandon magic or die.
Every time the fight ends with a weapon in my Mage's hand... which is just about every time.
The UI is annoying, there is lots of bugs, AI is wonky and Dragon attacks in towns are killing store owners and quests left and right. Those things are problematic, but the removal of Spellmaking is a fundamental design flaw that has noticeably hurt gameplay.