I am fighting the super mutant (not the boss), i hit his head with assault rifle via the VATS system, and yet he is only hurt a little bit (probably less than 15% of his hp) despite hitting it every single time, i eventually had to resort to using my jingwei sword all the time.

My small gun skill is above 80+ and i am on level 10. Thanks

Theres one super mutant in the germantown police station that is a overlord but is erroneously labeled "Super Mutant"...is that him? Otherwise headshots usually kill normal super mutants quite quickly for me and i have similar stats.

But some enemies like super mutant overlords, sentry bots and feral ghoul reavers dont seem affected by having 4 magazines of assault rifle fired dumped into their face. The reaver can even take 2 boxes of minigun ammo and not be affected.

Combat is really frustrating when a raider with a baseball bat is not even slowed down by a full magazine of 10mm to the chest, and crippling enemies's heads don't seem to do anything(they have no trouble finding you or attacking you...). Miniguns in fallout 2 that would kill most people in one pointblank burst are horribly ineffective now, people can just run through a stream of minigun fire all the way up to you and be midly injured.

Perhaps switch to a Chinese Assault Rifle. Or its unique, which is nice too. That said, I find hunting rifles or the Repeater, to be perfect for those guys in VATS. Try a rifle... less spread than the assault rifle. Perforator works really well too.

Thanks for all the comments, i will try that in the future. I thought the Chinese assault rifle is garbage, therefore i sell most of them once i find them.

its not even the overlords sometimes, i had trouble killing the Brutes or Master, if i encountered them, i usually just back off

If you do a little repair on one, you will find that it does higher damage than the assault rifle, in addition to being dramatically more durable. I still prefer hunting rifles (Ol' Painless) or Lincoln's Repeater, or even Blackhawk for those types of headshots. If you find the unique version of the Chinese rifle, you will be very pleased.

Why would you use VATS with an Assault Rifle just aim and either keep shooting him in the head, shoot his weapon out of his hands or switch weapons.

Because VATS allows you to fire much faster in a shorter period of time and the spread is much less.

Even with 100 small guns firing while in non-VATs has the bullets go everywhere. It is hilariously inaccurate and unrealistic. Assault rifles ingame cant even hit the center of the crosshair when firing down your average corridor.

Would luck had to do with it? Because i set my luck to 1, i thought it would be useless

Luck is definitely not useless in this game. Some perks require a certain luck level to get. You get better hits with a better luck, read: crit hits. I usually put my luck at 6, except for my current guy who is a 4. There is a bobblehead out there with Luck written on it. There are a couple of items that you can carry or don, that will artificially boost your luck. Next playthrough..... check it out. Read the list of perks in the manual and see where Luck fits in for your character.