Do you not have the expansion or personal choice?
I modded out the 24-hour in-game notification that psychically tells the character about Shivering Isles. So they have no way of knowing about it unless they actually see the island, which brings us to point two. My characters can't see far because I've modded infinite view distance out of my game. LOD is disabled and the fog is close to my characters (standing outside the sewer we can only see a few parts of Vilverin and can't see the land behind it at all. So far, none of them have gotten close enough
and wanted to go take a look at the same time.
We have more than enough things to keep us occupied in Cyrodiil though.