When I do quests, they seem to send me to this dungeon a lot even in different guilds. The Knights of the Dragon twice, the Fighters guild a few times, the Mages guild once, I done a number of quests there now. Is there something special there? Should I be visiting there without a quest?

You are sent there as part of the main quest as well (albeit I think it's an optional part). Nothing special about the place (aside from the fact that it's nothing like a castle). Just a popular location for dungeon delving quests.

Yea, for whatever reason it's a favorite choice for the quest generator :shrug:
If you get tired of it, you can always move to a different kingdom.