I think that they more or less formed the "far back-end" of the Empire. Given their location, they'd be comparatively well-defended; the land is pretty mountainous, with a lot of steep cliffs for shoreline, so an amphibious invasion would've been far harder than it'd be for Hammerfell. That, and the fact that the Bretons have a bit of elven heritage (or at least the resulting magical talent) might've influenced some of the Aldermi Dominion's decisions regarding them. Overall, I think they got ignored because they weren't directly involved in an invasion, unlike every other province in the land:
Spoiler - Hammerfell - Invaded by the Altmer, removed from the Empire, defeats invasion themselves.
- Skyrim - Not really invaded, but the main staging point for counter-attacks from the Empire. Also drawn into civil war.
- Morrowind - Decimated, invaded by the Argonians.
- Cyrodiil - Decimated, invaded by the Altmer.
- Valenwood - Usurped in a coup by the Altmer, turned into a vassal state of Alinor.
- Elseweyr - Tricked into becoming a vassal state of the Altmer. (Stealing moons isn't as easy as the Altmer made it look; generally illusion tends to be involved, and this was likely not an exception)
Summerset Isle Alinor - Source of the invasion of most the rest of Tamriel.
Black Marsh Argonia - Source of invasion of Morrowind.
Technically, Orsinium might be considered a province from the angle that it has its own native race, and it likewise escapes attention. (though really it's more of one of High Rock's 9 kingdoms)