That's exactly the point, your character may simply not be interested in going there. You might be playing a mage who is more inclined to help Valaste with some arcane research in an Ayelid ruin, for example.
Or another character you play next time might be more interested in what is going on on this island. If you don't do every single quest with your first character, it enhances the replay value.