This topic doesn't get brought up enough.
Aside from what others have already said; curses, traumatic deaths, being bound to an object.
Maybe as long as you are not bound to say, Hircine or Nocturnal. But even so, does one ghost (assuming a Nord) travel to Nirn from Sonvngard as they wish?
I'd imagine so. They just don't, cuz hey, mead and Shor and mead and Shor and mead and Shor and and and and mead.