Not the end of the world I know, especially compared to some of the glitches other players are experiencing. HOWEVER... it is getting extremely annoying. The first time it fixed itself, another time I was able to fix it on my own. I was mainly curious to see if other people had experienced this face fuzz glitch and if they had any knowledge of what was causing it in the long term. For anyone who is curious on how to fix it (or what seems to fix it):

-Go to the barber and have your face clean shaven.

-Save

-Quit Fallout 4 and/or reboot your console

-Reload and select your facial hair

If it doesn't fix it through those steps then rinse and repeat. If it doesn't work after another try, you might just have to give it some time.