I'd opt for the High Elf myself, mainly because the concept would require me to think further "outside the box" than a human character. The Breton would probably be easier from a mental standpoint, because his/her motivations would be closer to your own, and easier to relate to. The High Elf would present more of a challenge, but could potentially be more interesting.
Both the toughest and mot rewarding character I've RP'ed was an Argonian, and it changed the way I viewed humanity and its pre-conceived notions. Having to "unthink" all of the things that we take for granted as warm-blooded creatures, and all of the mammalian characteristics, and view the universe from a reptilian perspective, forced me to see a lot of things differently, and understand WHY people do the things they do.
Ultimately, it all comes down to how much you're willing to put into it, and how much you expect to get out of it. Either that, or some people just have more of an affinity for one or the other.
i'd vote for altmer. maybe give it a nice dark tan, sea-blown hairdo, and a pirate looking outfit complete with bandanna and cutlass. haha.