Wisps are probably the most incorrectly assigned soul level in the entire game. They simply are not Common level enemies due to their abilities, at least not when compared to the various other creatures. Then again, Oblivion has a much worse problem with soul levels than Morrowind had.
As for anyone saying they are easy kills... well, only if 1) you have the proper preparation and, more importantly, 2) you do not meet them in closed quarters. If the latter happens to you (such as many caves, for example) and you do not have an open room/cavern to move to quickly, you will be in big trouble because you cannot gain range to do distance attacks. This is excluding preparations such as having magic resistance items on, of course, which falls into provision 1).
I'll also point out the speed boosts that happen in my game. Yes, my characters move fast, but as they move faster and faster, the enemies also speed up. As I've said elsewhere, even the UESP wiki patrons have still not figured out all the parameters for speed. Since Wisps are one of the fastest creatures in the game, even a very fast character like mine will be in a goodly amount of danger if they run into Wisps in close quarters where they cannot create and maintain a distance (again, barring protections from magic attacks, of course).
Naturally, it also hurts that I play on max difficulty with 6x damage for the Wisps...
Xivilia are a piece of cake compared to Wisps, and Xivilai are Grand Souls... doesn't make sense, but that 's Beth for us.