Easy/Medium/Hard only affects your starting resources. Once you're past the hurdle of starting with less people/resources/seed crop types (or no seeds at all), which once you're familiar doesn't really take that long, it plays the same. The main long-term difficulty additions are the other settings - Mild weather vs. Fair vs. not-so-fair, Valley terrain (flatter, more space, easier to build on) vs. Mountain, and Disasters on or off.
Somewhat disagree. Don't start with orchards or herds, no. But the max 15x15 seed-field (cabbage, corn, whatever) only takes 4 workers, just like fishing wharfs, and 15x15 produces a lot for your 15-20 people. Hunters need 3 workers, Gatherers need...I forget. 2-4. Of course, most people start with the Gatherer or the Fishing (or both) anyway because they're easy to plop down anywhere/don't need to clear land first, but I start using field-crops the 2nd season (on Medium, where you're still given a couple of those). I haven't bothered with livestock, doesn't seem worthwhile.
I've played about 25 hours now. Been using Medium/Large map and Disaster off while I learn the town mechanics and wonder how hard it is to do 300+ pop. (didn't do tutorial or read the manual, more fun that way). My 2nd town got to 160 people and was doing pretty well/stable as I expanded but my layout was terrible so I started over yet again. 3rd time I already feel familiar with the population, town layout efficiency and food cycles so it's moving quickly and easily.
The main issue with this game is the population growth mechanic, as mentioned by Doubler. Kids who turn into advlts won't have their own kids until they have their own house to move into (with someone else). The problem is how fast do you build houses vs. death rate vs. kid-to-advlt rate vs. the fact at a certain age advlts can't have any more kids. Too slow/not enough kids born and eventually everyone starts dying with no one to replace, too fast and you can't keep up with the food/resource demand of all those kids+growth, hehe. So far seems like 1-3 houses a year works ok for very slow growth - at least when you have no Disasters to deal with.
P.S. - I believe the aging rate in the game is 1 season = 1 more year older - they numerically age faster than the in-game full year.