At the end of it all, is your financial return greater than the costs of doing it? Who knows. Obviously it's a risky venture, or else more people than Valve and the occasional indie dev would attempt it.
I think viability tends to be over- or under-estimated. Mac and Linux aren't going to be a huge source of revenue next to Windows any time soon, but it's not completely barren, either. Private contractors and companies have been porting commercial games and publishing them on Linux for well over a decade, now. id Software releases Linux versions of their games, and have done so since Doom or Quake -- not sure when they started Mac versions, but there's some of them, too. There really is a good number of commercial games available if you know where to look, and there is plenty of room for it to grow if properly nurtured (Linux has become very user friendly over the years, so the more well-known games Linux gets, the more people will take an active interest in Linux, which will drive demand for more games, etc).
Ryan Gordon, one of the more prolific figures who's ported some of the more well-known titles like UT2004, Serious Sam, and Prey, said in a recent interview (seen http://www.abclinuxu.cz/clanky/rozhovor-ryan-c.-gordon-icculus?page=1) that the biggest hurdles Linux has to gaining traction as a gaming platform is perception and misinformation, and I think that applies to OSX, too. Combined with no one to "speak for" Linux (like Microsoft does for Windows, or Apple does for OSX), it makes it that much more difficult to overcome those problems. It's not impossible, though. It's just not something you should give up on if you really believe in it. If you give up, you guarantee it will never happen.