What is wrong with co-op? It isn't like ignoring the quest compass / not using fast travel, the game will work fine n dandy if you don't use it. Personally I would love co-op, and I see no down side, it would hardly take any resources compared to some of the rubbish I see around here, anyways.
There are downsides. A multiplayer feature would require some pretty dramatic changes throughout the game, the most obvious of which is the interface. While it *might* be acceptable for only 2 players, having the game pause every time you open your inventory, change spells/equipment, use an item, check your status effects, etc, is not feasible for anything more. One player wanting to rest for 5 hours; is the other one forced to nap too? Will the layout of dungeons have multiple players in mind, to keep them from being a third wheel in narrow corridors? Will there be multi-person puzzles, basically either penalizing you for going solo or making a partner feel pointless with their absence? How will the story deal with Hero and Friend? A great many game features will have basically two options: force the players to be chained at the hip and do everything the same way at the same time, or change the way they work. As well, resources would be considerable. Not in development space, but in time. TES games already tend to be riddled with bugs and exploits, a nearly-unavoidable side effect of having a very large, open-ended sandbox world. Linearity lets you test along very specific gameplay lines. Adding co-op creates the problem of whether things work while multiple players are involved...regarding every single thing in the game.
So no, the game is not left unaffected if you don't want to use it. There's also the simple matter of focus; you can pour all your resources into a tennis game, and make a great one. Or you can make a mini-game compilation with lots of different sports, but the tennis aspect will not be as good or as in-depth as if it were your only focus. Changes have to be made to the game to accommodate co-op, and those changes will almost invariably have to move away from the focus of "ideally designed for singleplayer." Some games, like first-person shooters, easily adapt to multiplayer, but TES has been designed solely around the singleplayer experience. For people only interested in the singleplayer, the addition of multiplayer is a stamp that all but guarantees harm to the gameplay experience they wanted.