Your memory hasn't failed you. The UESP even mentions this: "As one explores the larger dungeons the detailed map may become increasingly confusing."
But that would be true no matter what map system you use.
Anyway, while I admit that Daggerfall's map can be a little unwieldy sometimes, I've found it much more useful than the map systems of the other games, which can't even differentiate between multiple floor levels.