Hypothetically speaking, what would happen to the Mundus if the Amulet of Kings were taken and used for an extended period of time in the Shivering Isles (ostensibly, this applies to all of Oblivion, but for the purposes of this experiment, the Shivering Isles)? What would happen to the Isles? Would the Amulet even work as it is powered by an Aedra and their influence falls away rather sharply outside of Nirn?

I would say, likely, nothing?

probably nothing since the amulet of kings no longer exists. but for you question it did exist and you did bring it there. probably nothing

I believe the amulet has already spent quite some time in Oblivion during the Imperial Simulacrum. Nothing much happened. Jagar Tharn was actually counting on nothing happening, as if the Dragonfires had gone out he wouldn't have been able to keep up the illusion, no matter his skill.

The AoK is the stone to White Gold Tower. The Tower is in Mundus, and as such it really has no use in other planes, especially considering the Shivering isles itself is a much more effective barrier system then white gold alone. Seeing as the whole need for an heir was symbolic, and the real trick was jsut bringing the amulet tot he temple of the one, removing the amulet from Mundus would simply de-power White Gold, which as we know is not a good thing for Mundus.