After finally adding the 5th level to the big leagues perk, my damage on melee weapons (not unarmed) went down significantly. Something like 180 to 102 on my stunning super sledge, for example.
I've tried restarting and going back to multiple saves to try leveling the perk and making sure it is the perk that's doing this, and I'm sure it is.
Maybe it's a problem with the math the game does for the final level of the perk? I have no clue.
Edit: After testing it seems to only be a bug with the numbers shown in the pip boy. My actual melee damage was increased rather than decreased.