I agree fully with this. I even agree with the notion of acquiring and using all kinds of raw materials, if it fits with the roleplay. For instance, a mage might decide to build a house or two (Hearthfire), and acquire all the materials the "hard way." That's a lot of mining, smelting, and smithing of nails, hinges, fittings, locks, etc., and it's bound to drive one's smithing skill up. (But it doesn't benefit in any way from perk points.)
I'd say that as a general rule, I'd want to put perk points into my character's main offensive and defensive skills first.
So, for a sword&board guy, I'd put points into 1Handed and Block, and improve those perk trees as skills rise. Get some points into armor. Try to keep smithing perks balanced with the character's level; you don't need glass or ebony skill at level 10. (The most important perk in Smithing, I think, is Arcane Blacksmith.)
For a mage, I'd be putting points into Destruction, Alteration, and Conjuration. A few points could go into Restoration, if you use wards. The Illusionist might go heavy on that skill, and only perk Destruction enough to have Runes and one augmented damage branch. The exact approach here is an individual preference. The idea is to have both a strong attack and a strong defense; anything else is relatively unimportant.
My whole point is, when possible, to work on survivability and attack first, and then worry about the Crafting when you've got points to burn (such as when you don't have a high enough skill to qualify for that next "important" perk.