Make them vary in size, from little to huge - it doesn't matter. It will contribute to the immersion and the nature of Skyrim will feel so much more realistic. But yeah, I voted for 'Make 'em Bigger', because trolls are essensially made to be huge and they should be huge in this game... and hard to kill... and rewarding loot.
http://www.youtube.com/watch?v=us9ox2pMWzw
But by that logic, there would be such things like little lions and big lions. Real animals don't really have much size variance among the same species. I mean, it could be slightly bigger, but you couldn't have a "realistic" variance between 5-foot and 18-foot trolls. If there are things that are that big, it'd be because of some freakish mutation. I'm still unable to get over the difference between Fallout 3's normal super mutants and the super mutant behemoth.