I've heard the Sami are quite short in general, and the ones I've met always have been - unfortunately, I don't know enough Sami people to give an accurate guesstimate at what their average height might be. And I've never seen an actually accurate source say they're short, it's more on the basis of hearsay. I'm unable to find any
reliable sources to back up their shortness, though, but http://www.cedarseed.com/fire/humantypes2.jpg pops up with a quick Google search. Anyway, the point is that if the Sami are indeed quite short and have lived in the extreme North (as they have), that makes the "taller because of colder climate"-argument less probable.
It's true that the Scandinavian countries have been "isolated" for a long time, so a narrower gene pool might be what caused it.
At 5'2'' I'm below average height by Finnish standards.
Damn. I should move to a country with a shorter average height to make myself feel taller.