instead of stopping blocking they could make it so you're staggered more easily as you attention is divided, you cant knock people down or whatever one handers do, you don't get specialization bonuses from perks and with one hand free you can do a grab stab move (bonus with dagger) or if they block a push to stagger them instead of a slice.
There was a previous thread on this a few days ago. Some people suggested taking reposite, parry, only taking partial damage if you try to block instead of full damage. I agree that there should be something else done.