seems to suggest they are fairly large, the major towns of each hold, of which there are 4 (I haven't been able to conclusively figure out where that last hold is BTW). Going by this: "while the other 4 have smaller, what we would call, like a town. A little bigger than but like that." So yeah, I guess they have a castle of some kind, if they're where the Jarl of the holds live. Maybe they'll be a bit like Caldera in Morrowind.
The fourth must be the Tundra marsh hold (i think it says marsh on the map). You know the Hold to the east of Solitude and west of Dawnstar. The hold town there might be Dunstad or Stonehill/s.
Yeah Todd speaks a little confusingly sometimes. Probably having to go through that many interviews.