First we should assume that a races strengths and weeknesses vary region by region. We have no reason to believe that the village idiot/local drunkard/paranoid gravedigger in Vvardenfell or Cyrodiil is a good representation at all to the Nord populus in Skyrim. How long do you think an intelligent Nord would last in Redoran territory?
Second, while "Intelligence" is a singular attribute, the Intelligence we are discussing is hardly singular. Even in the Elder Scrolls universe we have to assume that there are different types of intelligence. I.e., Alchemy is Naturalist Intelligence, Speechcraft is Interpersonal Intelligence, any form of combat, along with Athletics and Acrobatics, would be Bodily-Kinisthetic Intelligence, and etcetera.
That said, assuming they do find magic pointless, bear in mind that physical combat is a great deal easier to use in TES, and to use very effectively and practically, than magic.
On another note, The Norse weren't unintelligent by any of the stretch of the imagination. They pillaged because it was survival. Farming can only go so far in the conditions they dealt with. They also designed some of the best sea vessels of their time.