If an attack is less powerful than target's DT, the target will still take 20% damage from the attack. So 5 against 10 will do 0.2*5=1 damage.
If an attack is more powerful than target's DT, the DT is simply subtracted from the attack. So 15 against 10 will do 15-10=5 damage.
Pretty simple, but very effective.
Oh, and if you play on PC with mods with items that have Damage Resistance, the resistance is calculated from the DT result. So with DR50 the target would take 0.5 and 2.5 damage in the above examples.
IIRC, no vanilla item in New Vegas has Damage Resistance, so if you are on console or without mods, ignore that