inaccurate float math : V

» Tue Nov 19, 2013 11:56 pm

Hmm, anyone experienced something like this before? My float multiplications are coming out wrong, and since eventually have to cast them to an int, it has caused some problems. You would think 100.0 * 0.80 is simple enough not to equal 79.999992, but I guess not?

Spoiler

[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotMGEFBase[5]  ====================> 100.000000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotFactor       =====================> 0.800000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: mag = hPotMGEFBase[5] * hPotFactor ==> 79.999992[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: dur = hPotDuration  ....   dur =======> 2[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: MAG = ((mag / dur) as int) as float ==> 39.000000[11/18/2013 - 11:08:46PM]   [11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotMGEFBase[4]  ====================> 25.000000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotFactor       =====================> 0.800000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: mag = hPotMGEFBase[4] * hPotFactor ==> 19.999998[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: dur = hPotDuration  ....   dur =======> 2[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: MAG = ((mag / dur) as int) as float ==> 9.000000[11/18/2013 - 11:08:46PM]   [11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotMGEFBase[3]  ====================> 50.000000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotFactor       =====================> 0.800000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: mag = hPotMGEFBase[3] * hPotFactor ==> 39.999996[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: dur = hPotDuration  ....   dur =======> 2[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: MAG = ((mag / dur) as int) as float ==> 19.000000[11/18/2013 - 11:08:46PM]   [11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotMGEFBase[2]  ====================> 75.000000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotFactor       =====================> 0.800000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: mag = hPotMGEFBase[2] * hPotFactor ==> 59.999996[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: dur = hPotDuration  ....   dur =======> 2[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: MAG = ((mag / dur) as int) as float ==> 29.000000[11/18/2013 - 11:08:46PM]   [11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotMGEFBase[1]  ====================> 100.000000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotFactor       =====================> 0.800000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: mag = hPotMGEFBase[1] * hPotFactor ==> 79.999992[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: dur = hPotDuration  ....   dur =======> 2[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: MAG = ((mag / dur) as int) as float ==> 39.000000[11/18/2013 - 11:08:46PM]   [11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotMGEFBase[0]  ====================> 150.000000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: hPotFactor       =====================> 0.800000[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: mag = hPotMGEFBase[0] * hPotFactor ==> 119.999992[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: dur = hPotDuration  ....   dur =======> 2[11/18/2013 - 11:08:46PM] :::::::::::::::::::::::::::::::::::::::: MAG = ((mag / dur) as int) as float ==> 59.000000

» Tue Nov 19, 2013 2:05 pm

hey, it's ck, rotate 30° + 30° + 30° == 90.00001°. any other questions...? ;-) )

» Tue Nov 19, 2013 7:45 pm

Those are standard floating point precision errors. They occur in every programming language. There's nothing wrong with Papyrus or the CK here. When you turn float values back into an int make sure you round them instead of just doing a basic cast. So use "(value + 0.5) as int" instead of just "value as int" and you should get the values you're expecting.

» Tue Nov 19, 2013 2:22 pm

Strange as it is this is expected behavior of floating point numbers. Storing a float in binary is actually pretty complicated. Papyrus must be doing something a little strange since an IEEE-754 float would be 80.0000011921 since 0.8 is stored as 2^-1 * 1.600000023841858 (really) and 100.0 is 2^6 * 1.5625.

There are a few ways around this. You could stick with integer math. 100 * 0.8 = 79.9 <=> 100 * 80 = 8000, 8000 / 100 = 80 Or you could add a small number to your result before casting it as an int 79.999992 + 0.0001 as int = 80