I can totally picture 2 orcs concentrating and hanging over a chessboard.
Hahaha yes! Very good visual for a Monday at work, thanks!
To the subject, I don't think Tennis makes sense - but what Does make sense is that some kind of games take place (outside of the arena). This isn't Rome, and the inhabitants of Tamriel would likely be engaged in some kind of sporting event or typical gaming. Everything from kids throwing dice in the streets to gambling to more organized competitions might all be common-place in the era.
I think it would add some spice to have some kind of game included - it could add lots of conversation topics in taverns and might even spawn some quests for the player in which one team things another is cheating or throwing games, or when star players suddenly vanish, etc, etc. It would definitely add to the game, but I don't think that this idea is Necessary for Skyrim to be successful!