I think I have the defining characteristic of a casual game.
Before that though, a disclaimer: I have nothing against casual games and never use casual as an insult.
When I sit down at my game computer there's lot's of things I can do.
I can open my RPG menu and take up my Oblivion character, or run CJ through the streets of San Andreas, or guide my crew through the world of Baldur's Gate. Whichever I choose I will have to think a bit about where I left off. Or I could start something new, maybe Skyrim or Witcher or Fallout.
I can open my Total War menu and either start a new campaign somewhere (probably not) or figure out where my Scottish empire stands in M2TW and see what progress can be made there. Strategy menu; I have a Civ4 campaign that's probably easier to abandon than figure out what's going on there because I haven't played it in so long. Econ Sims menu; I think I would start from scratch if I played Port Royale, and I haven't played anything else there since I rebuilt the machine so no saved empires to pick up.
In combat games I have a M&B campaign where my Vaegars are about halfway to turning the whole world green, and a Freelancer game that's barely started.
Then there's Chess. When I play chess against a human opponent it's about half a step short of a fist fight as far as my commitment to winning is concerned. I can list every person I've encountered in my fifty year life span who forced me to admit they were a superior chess player, and my teeth will grind on every one. But Chess on my computer is my 'casual' game.
It's the only game I can play where I do not have any 'where did I leave off' to think about. I start a game. I play it through. No save. No remembering where I left off. If I play again tomorrow or next week or take a year off, coming back next time will be exactly the same. I think that's what makes a 'casual' game.
Agreements? Better definitions?