Weaker how? And, what gem?
Weaker: -45pts to Health/Stamina/Magicka
Gem: The soul gem that houses the taken part of your soul will be able to be retrieved.
All stats will be back to normal and you can re-enter the cairn anytime without repeating this step.
The weaker stats don't really matter much in the Soul Cairn unless you have zero ability to defend yourself.
You've made it to the portal, though, so you'll be fine. No worries.
Edit: Hiyah, Wulfie
