I started an unarmed build and am quickly figuring out how annoying this is. Outside of power armor it works okay, but since you cannot use unarmed weapons with PA equipped there is no easy way to switch to it. You have to open your pipboy and manually unequip your weapon.

The fact that power armor arms come with mods to increase the unarmed damage makes it seem like unarmed combat is very intentional. Yet, it's really annoying to get to in combat. Sure, a mod can fix this, but it really seems like something that should be in the base game.