So I took a quest to kill some bats in a house.

Killed the bats and returned to the guild. But when I select the questgiver and click "talk" or "get quest" nothing happens.

Is this a known bug or is something else going on?

I'm not quite sure what it is, but I've encountered this, as well. It's probably a bug. Daggerfall is infamous for bugs.

buggerfall can be a *****

Have ya try using http://www.uesp.net/wiki/File:DFQFIX.ZIP?

You have to kill ALL the bats/rats. There is usually 7 of them, sometimes 6. If the quest giver doesn't respond to "talk" there will be more vermin to kill if you go back .
(I'll usually do that quest a dozen times a game and never found it bugged).

Keep loitering inside the house till you get a message saying they're all gone.

Hmm... ok.

There were like two, but it was literally a one-room house. Should I go in there and loiter for a few hours or literally sit there?

Yeah, just loiter inside. They should pop out at a rate of one every hour or two. Just loiter till they're all gone.