You can swim to it, at least you can by swimming from the spot of the bomber in the lake. Note that I did this to kill everyone there, so I don't know if that messes up any quests.
I tried that, but once I got ashore, I couldn't find ANYWHERE along the Lake Mead shore or Colorado River East shore where I could ascend to the upper plateau where Fortification Hill is located. The ONLY way I have ever been able to get to the fort is to wait for Caesar's invite, go there, and then the fort becomes a known location that I can fast travel to and from.