Correct me if I am wrong, but I thought armour in TES was % based? So at 85AR (max) you reduces damage by 85%. Which would make that 25 v 56 damage thing, against a 20AR enemy, turn into 20 v 44. Rather than 5 v 36.... Which to me seems fine for 6 perks investment.
As far as magic skills go, how do you know it doesn't
ALSO effect magic casting cost? Maybe, a master of Destruction will, by virtue of their skill, be casting a 100 magicka spell at 80 (20% reduction), while another master who specialised with perks will be casting it at 40 (50% reduction from the perk, followed by 20% from skill). EDIT: A point has also been raised that leveling your skill may increase the power of the spells themselves.
You cannot say this is not possible because
you do not know.
All you are doing here is saying "hey, what if it was like this? I would be annoyed!", and not even giving good arguments for that

The Smithing skill is a good example people doing this. There is a lot of presumptions flying around that it is useless after the low levels without perk investment. Well, that is one interpretation. However, it could well be that
irrelevant of perks you can improve weapons & armours condition all the way up to the highest tier of material, but the Smithing perks simply means that you can 1) personally craft those items 2) and then
DOUBLES your improvements.... you know, as the text says. So normally you increase damage/AR by 15%, now you are improving by double at 30%.