Curiously, the spirits I've seen all appear to recall their lives quite well, it's what happens afterwards that starts to get fuzzy. Perhaps the loss of memories comes the fact that when a spirit is reborn, it is placed in a vessel which is initially incapable of retaining the knowledge that the spirit has gathered. Kind of like deflating a balloon to fit it through a small opening. Whereas Daedra are presumably reborn in full-sized, functional bodies.
re location - people talk of the spirit realm - that indicates a place. perhaps it is co-terminus with the more solid mundus or whatever?
The spirit realm is not a location, but rather as 1999 noted a spiritually percieved environment. If somebody had levitated high above the IC while you were fighting Umaril they would have noticed nothing out of the ordinary due to them not 'being in the spirit realm', ie, not being dead theirself - afterall, you had to die to get there...
An individual cannot travel there physically since you're not really going anywhere, only in spirit or in mind. Notice that your body stayed in Garlas Malatar when you were fighting Umaril the second time, it was your spirit that entered the realm, the realm of spirits. Umaril's destination wasn't some "spirit realm" whose location is in the sky above Cyrodiil, his destination was Oblivion, you cut him off before he could get there, your spirit intercepted his spirit in the sky over Cyrodiil. The fight could have hypothetically taken place anywhere between Mundus & Oblivion. At that point you were pure spirit, which is the same thing as being in the spirit realm.
The spirit of Umaril should not be confused with a ghost. A ghost is ectoplasm animated by a soul, Umaril was just the soul. You can see a ghost, you can't see the soul, you can capture it, but you can't interact with it unless you are in the spirit realm and you can't be in the spirit realm unless you are a spirit.
One other thing we can learn from the encounter with Umaril is that a body can only kill another body, but that won't directly kill the soul (afterall it can still be trapped). However, a spirit can kill a spirit, utterly annihilating its existence (which sorta makes me wonder if a shehai could manage it).
Edited for clarification.