The point isn't that you're spending the perk on Smithing, its that people are spamming iron daggers to get their levels up to the point where they can get ebony level armor early in the game. You DON'T need to do that.
Case in point, I'm playing as a warrior who smiths, but I'm keeping my smithing around the level of my heavy armor and blocking, which are right now around 30. Therefore, my smithing is at thirty and I'm able to smith dwemer armor.
My issue with crafting is a bit more deep seeded than that though. I don't really mind the "rate" of XP gains, I mind the fact that crafting, as an overall skillset, results in something far more powerful than leveling "normal" abilities. If you cheese out smithing/enchanting the right way, you hardly need to buy any armour perks, as you can hit armour cap just through the quality of the base armour and natural armour skill. You can squeeze out 90-100% mana cost reduction on spells for near infinite magic. You can maximize your stealth ability. All of this is contained in 3 skills, while normally these things cover the other 15 skills in the game. Crafting alone is crazy in how it supplements/replaces so many other talents. Its no contest that crafting is a superior method to improving your character.
It literally is a money tree as my previous example went. Why would you not own one except in order to hinder yourself? And why should I be forced to hinder myself in order to have different and "fun" experiences each time I make a new character?