Left Column - Your Inventory
Right Column - NPC's Inventory
You can go through the columns like you do with your pip boy to narrow stuff down that you want to buy or sell. Ex. Aid, Ammo, Weapons
If you want to buy something from an NPC, go to his/her inventory and click on the item. I giant arrow will appear pointing to the right, with a number of caps. That is how much caps you will have to pay.
If you want to sell something, click on an item in your inventory, and an arrow will appear pointing to the left with a number of caps. This is how much caps the NPC will give you for the item.
You can sell/buy stuff in bulk and it will add up in the center.
Also if you buy something you don't have enough caps for, the arrow with become transparent, same applies if the NPC doesn't have enough caps to buy your stuff.
To cancel the transaction, hit back, whatever button you use, and "Cancel Transaction" will appear in the center of the screen with "Yes or No". Click Yes if you want to cancel, No if you don't.