It doesn't make sense? The potions are practically poison. They would kill him if he drank too much. The "meditation" state is probably what allows him to drink more than one potion. That's a system of balance with a good lore reason to fit everything in perfectly instead of just being an unpsoken gameplay mechanic rule.
Well, actually, the reason the potions in The Witcher are toxic is the fact that they are made of materials highly poisonous to anyone with unsuitable metabolism. Witchers (and some other magic-touched beings, like wizards) are 'mutants' whose organisms function very differently, which is what allows them to gain benefits from the potions they brew. To any normal person, these elixirs would not only be useless (incompatible metabolism unable to trigger the same effects), but in most cases lethally poisonous as well. A witcher's organism is largely resistant to these toxins (albeit not completely) which is the only thing that allows them to endure multiple imbibings without passing to the Other Side five minutes later. All that meditation is responisble for is, in Geralt's case, accelerated neutralisation of the poisoning, which otherwise would occur over time, but witchers can also exert a measure of deliberate control over their metabolism while in trance, so that's a much faster solution.
Now, I don't think there's even any point in comparing the 'potion systems' of both universes. The world of The Witcher is pretty low-magic, dystopian and heavily folklore-influenced, as opposed to the epic, overblown high fantasy of TES. In The Witcher's setting, magic is unpredictable, feared, and 90% of the time utterly beyond the reach of most common people, who are forced to rely on
conventional medicine (no doubt including leeches, herbs and prayers) in situations to which the inhabitants of Nirn respond by
taking a short trip to the nearby alchemist's shop or the wizard living next door instead.
Instantly working and harmless potions would be against the Lore of The Witcher, while 'Swallow' in the world of TES would be considered merely a dangerously botched attempt at alchemy. I suppose it's just like expecting the mechanics of D&D to work with the setting of WoD, or vice versa - technically possible, but completely unfitting.
(And to disperse any doubts, personally, I'm on The Witcher's side of the barricade, always having enjoyed that pseudo-realistic, strategy-demanding system much better, but apparently the TES tradition is against me, so I will refrain from pointlessly raging against it as to avoid adding even more fuel to the already enormous flame.)