If you store these somewhere, are you going to keep your extra stats?

Yep, you keep them. You can even see most of those effects in your Pip-Boy's Perk list.

Just for the reference, you also keep your Companion Perks (the ones you gain after maximizing their affinity) even if you dismiss them.

You can do whatever you want with them and still keep the perk. Sell them to a vendor, blow them up with mini-nukes, or whatever you want and you will still keep the perk. The only way to remove the perk is to use console commands. For example, player.removeperk 61686 will remove the Sneak Bobblehead perk.

I love putting them on magazine racks in my crib. I do not know why.

As others have noted, you can see most of them in you pip boy somewhere...