Anyone know? The door is inaccessible and it won't let me just swim up onto the shore bank.

I had this problem too, and I found the way, it's alot easier than I thought. Follow the river south and you'll hit a Legion camp with docks called something like 'Cottonwood grove' or something, and there will be a guy on the docks, talk to him, and he will take you by boat to the Legion Camp.

Hope that works, although I did kill nearly every legion guy there before.