While that's true, I'll have less perks invested into combat, thus making me less efficient in combat.
You'd be surprised to find out that 100 alch/ench/smith can result in a stronger "base combatant" than fully speccing in 3 combat skills.
For example, I took a base character, advskilled all the crafting to 100, and added the needed mats. I created a "perfect" set of daedric armour/shield/longsword/ring/amulet, and enchanted it with fortify heavy armour and fortify one handed (all of this done under the effects of maxed out alchemy potions).
On its own, the crafted gear resulted in about 690 armour, 140 attack.
Then I took a base character, advskilled Heavy Armour, Block, and one handed, and equipped with "base" daedric gear. 490ish armour, 40 attack.
Crafting > skill. Or rather, it is a matter of having the coolest toys, not a matter of knowing how to use em.