Well, Requiem is a total Overhaul and Difficulty mod, sort of a Hard Core, Iron Man, Role-Playing thing which makes everything more deadly and changes almost every aspect of the game mechanics. I don't actually have it, though.
Phantasmal Killer is an AD&D Illusionist spell going all the way back to First Edition. Basically, it creates a phantasm in the mind of the victim who is struck with horror and sees the most terrifying, hideous thing he can imagine - his deepest, darkest fear. He flees in terror (in his mind) as the monster pursues and if the creature hits him he dies of shock and fear. The monster is invulnerable to attacks and spells and can pass through barriers like a ghost (IE, no collision). The only chance he has is to disbelieve the phantasm or, in Skyrim, Resist Magick. It's only 4th level, so probably Apprentice in Skyrim. Maybe Adept.
In Skyrim, you could probably just have the victim enveloped by a shadowy shape and then save vs magick or flee in terror and ten seconds later drop dead. Only he can see the monster, after all.
Skyrim Illusion spells are basically Fear, Calm and Frenzy, so there's not much you can do. They are mostly Charm-type spells. But in AD&D you had audio, photonic and chromatic illusions, phantasms, shadow magick and the invisibility family as well as others which mimicked Conjuration and Evocation (Destruction).
I'd love a decent set of Illusion spells for Skyrim.
The main True Illusions relied on the victim believing the illusion, and the damage was illusory (imagined, not real).
Phantasms could cause phantasmal damage (where the mind affects the body and real damage could occur, like stigmata or dying of shock), while Shadow magick spells were quasi-real and did real damage, although not as much as a real summoned monster or Fireball. Proper illusions would not work in Skyrim except to act as decoys that cannot do actual damage.
EG an illusion of an Orc would be a summoned Orc that doesn't actually do any damage but would distract the enemies as they kill it.
A Shadow Monster Orc would do some real damage while it acts as a decoy.
An illusion of a Fireball would turn them hostile but not really burn them.
A Demi-Shadow Magick Fireball would do some real damage but not as much as a Destruction Fireball and would not benefit from perks like Augmented Flames and Destruction Dual Casting etc. Illusion perks WOULD apply.
So for low or medium damage Illusions, the easiest thing is to borrow from the Conjuration and Destruction school but call them Shadow Monsters (20% max hp), Demi-Shadow Monsters (40% max hp), Shades (60% max hp), Shadow Magick, Demi-Shadow Magick and so on.
But if you want some other ideas:
Colour Spray (Novice)
A flash of vivid rainbow colours hits one NPC or monster and causes a sensory overload that stuns them (stagger) for 5 seconds if they save and if they fail to save vs magick they are unconscious for 10 seconds and (when they wake up) blinded for 10 seconds. It doesn't do actual damage, but you should be able to hurt them with your dagger and then cast another spell.
Rainbow Pattern (Apprentice)
Something like the Eye of Magnus, a floating, swirling, sphere of scintillating colours, is created and anyone within range has to save vs spell (resist magick) or is Fascinated (or paralysed) for as long as they look at it. I don't know if you could make them follow it as you move the sphere so they walk off a cliff, though.
Phantasmal Killer (Apprentice, maybe Adept)
I mentioned already.
Prismatic Spray (Expert)
A 70 foot long shimmering, multicolored ray of light 15 ft wide at the end spews forth from your hands and any enemy lower than level 12 is struck blind for 20 seconds regardless and all enemies are hit by one of the coloured rays, with a 1 in 8 chance of being hit by two rays at once.
1. Red 20 points of damage, save vs. magick for half.
2. Orange 40 points of damage, save vs. magick for half.
3. Yellow 80 points of damage, save vs. magick for half.
4. Green Save vs. poison for 20 points of poison damage or die
5. Blue Save vs. magick or be turned to stone (paralysed for 120 seconds).
6. Indigo Save vs. magick or go insane (Fear or Fury for 120 seconds).
7. Violet Save vs. magick or be teleported to Sovngarde.
Mislead (Adept)
A simple double of yourself is created which has a simple script to interact with the enemies while you are turned Invisible and can run away from the area or do whatever you want. (Invisibility True if you can make it).
Project Image (Adept)
A double of yourself is created which can use weapons and cast spells and is the same level as you with the same weapons, skills, perks and spells, and will fight to the death in a Frenzy against any hostile creature or NPC. You are not invisible and have to concentrate on the illusion, though. So if you do anything else, draw a weapon, cast a different spell or get hit by an enemy, the illusion ends.
Weird (Master)
Like Phantasmal Killer, but affects ALL enemies in 30 foot radius and even if they save vs magick they are paralysed for 5 seconds and lose 20 points from all combat skills, then flee in terror. Otherwise, they die.
I also like the suggestion that a spell crates a double of the enemy and it has to fight itself. Maybe call it Hostile Imposter or Doppleganger?
But an Illusion spell that exists in all fantasy games, including Daggerfall and Morrowind, but not Skyrim is Improved Invisibility (Invisibility True), where you do NOT become visible by opening a door or drawing a weapon or even when slashing something to pieces.
If you could make THAT (I've looked and failed), I'd try your mod just for that spell!
~.~