So I restarted the MQ and I don't feel like waiting for Baurus to SLOWLY walk through the sewers so I need to know, can I just go to Lake Arrius Caverns when the book quest starts and skip this bit?

Nope. Pretty sure you have to actually get the 4th book before you can advance the quest. You don't have to go back and talk to the librarian, though. Once you've got all 4 books you can go to the glowing tomb thingy to finish that quest and then you can go to Lake Arrius. If you go straight to the caverns you won't be able to get in. Unless I'm really, really badly mistaken.