Lets say i cast weakness to magic 100% and weakness to fire 100% on a monster, after which i blast the monster with a 30 points fire damage spell....
First the weakness to magic will magnify the weakness to fire to 200%.
Then when you cast the fireball (assuming the weakness durations are long enough) then the monster will have a 100% weakness to magic and a 200% weakness to fire.
The 30 point fireball will be magnified to 60 points by the 100% Weakness to Magic, an then again to 180 points by the weakness to fire. For a total of 180 points fire damage. (the actually order that the weaknesses are applied to the fire ball doesn't matter).
If however you cast a 100% Weakness to Fire and then 100% Weakness to Magic then the effect would only be 120 points fire damage, because in that order the Weakness to Fire would not be magnified.
EDIT to answer second question:
...
which means casting two 100% weakness to magic spells on a target would give the target 200% weakness to magic?
Yes 200%. But it is not stacking. The first case magnifies the second cast, and then replaces if they are the same spell cast twice.
But if they are two different spells with different names but the same 100% Weakness to Magic effect. Then casting one then the other would result in 300% Weakness to magic. (100 from the first cast and 200 from the second, and since they are different spells both stay in effect).