I know that limiting players to the main schools of magic is a way of simplifying game mechanics, and that there are greater magical feats to be found outside the games. But in the grander scheme of things, what sort of limitations are in place regarding what sort of magic can be used?

For example, in the Witcher series, practitioners of magic can use artifact compression, where they take a living person and shrink them down into a jade figurine for transport or concealment, and can restore them later. Or using polymorph spells in Warcraft, or blood magic in Dragon Age. Is there anything in the lore that states these more elaborate sorts of magics can't happen?

One thing to note is that the idea of schools of magic originated with the Mages Guild and is more of a way to organize spells by their effects than anything regarding how they actually work. In short, the existing schools are more or less arbitrary. Exactly what the limits are is never elaborated on, but the Psijics do practice forms of magic that are unrelated to the Imperial schools and apparently involve direct manipulation of nature rather than magicka.

A mage in Oblivion said that spells couldn't be permanent, but also demonstrated that you could make them sufficiently long lived that it didn't really matter.

Polymorphism in Tamriel is a...nasty affair. It tends to be associated with dangerous hedge-magics, and are less "I'mma turn into a bear" and more becoming some twisted abomination like Hagravens. Then there's the Wild Hunt, and that's infinitely more disastrous for everyone and everything remotely close to one. But as far as limitations go...there's nothing I'm aware that magic couldn't theoretically accomplish, in a sense. Now its worth mentioning that most Mages in the entire setting will never ever see that kind of stuff. Even in large group, extraordinary feats are a bit of a rarity, and the likes of Shalidor, Mannimarco, the Psijiics are going to be extremely rare.

That said, there's nothing in Tamriel that can't be considered magical to begin with. Its not spellcraft, but things like Pelinal running amok, Hrol's hill fetish, everything about the Dwemer, godly interventions are all usually magical in nature.

Not something we actually have any exposition on, aside from Telvanni's habit of self-experimentation to increase their own lifespan (which could be taken as blood magic in other settings, depending on the definition), nothing is ever cited as ever being different then normal magics, including things like blood sacrifices. Doesn't really seem to stand out as anything particularly special. I have a few headcanons regarding it, but that's really all they are.

Magic can be quite varied and as mentioned above the schools were made so that students of magic can learn it faster (though I believe the Mages Guild adopted the Shad Astula curriculum). Take for example Brelyna Maryon, a Dunmer mystic and a student at the College of Winterhold. She asks for your help and once you accept she casts spells on you which go a bit wrong and you are turned green, or into a dog and a cow. So there's your polymorph spells in a way. What we see in the game is just a representation of the magic that exists in the series