You don't need to be large city to be a capital. Hell, you don't even need to be the largest city in a territory to be a capital.
Look at Pennsylvania - biggest cities in descending order by population of : Philadelphia, Pittburgh, Allentown (on Saturdays during football season, University Park), Erie, Reading, Scranton, Bethelehem, Lancaster, and in 9th place and capital of Pennsylvania: Harrisburg.
How about New York? New York City, Buffalo, Rochester, Yonkers, Syracuse, and in 6th place and capital of New York: Albany.
How about the United States of America? I would go down the entire list but the first 10 are (and off the top of my head, so I'm not sure exactly): New York, Los Angeles, Chicago, Houston, Philadelphia, Phoenix, San Diego, San Antonio, Dallas, and I think either Austin or San Jose...
Washington DC is somewhere in the mid-20's.
So they might be towns, and they are the largest in their holds, but even if time progressed and they grew but other towns grew faster and became larger, they would still be the capital unless the Jarl decided otherwise.