Is it the name? Daggerfall, Skyrim, Redguard, Online and Infernal City are all very different products and all have "The Elder Scrolls" printed onto them. But what's in a name? Is anything with "The Elder Scrolls" written on it an Elder Scrolls product?
Is it the gameplay and mechanics? There are certain elements we came to associate with the Elder Scrolls series, like an open class system, level progrssion through skill progression, action oriented combat, etc. But then, we have seen many features that seem essential come and go. Arena had experience based leveling, Oblivion and Skyrim reduced the number of skills and the latter even got rid of any kind of class. The combat system changed from drag'n'drop with a chance to miss to click-to-hit. Redguard, Battlespire and the other games not belonging to the main series were completely different in gameplay.
Is it the lore? CHIM and Lorkhan, Leaper Demon KIng and Vehk, Moon Sugar and Guar, and all the other stuff that gets talked about on the lore forum. The Elder Scrolls series managed to differentiate itself from the more generic fantasy worlds by subverting or innovating common themes (dwarfs aren't dwarfs, dragons are world eating angels, magic through shouting) and adding unique elements (Khajiiti sub-species, CHIM, Sheogorath). One could argue that a fantasy game is identified by the world it is set in and thus Elder Scrolls must be defined by its lore. But most of the lore was made up between Daggerfall and Morrowind and the following games didn't shrink back from changing the lore for whatever reasons. If lore is that changeable, can it be used to distinguish an Elder Scrolls game from another fantasy game?