March 2, 201115 yr Commercial Member I don't know how many different ways I've tried to resolve this but below are just two of them. The problem is that even after rounding I keep hitting the binary division problem where I end up with a long tail of nines or a long tail of zeroes plus a one. This is making one or two autopilot calculations barf pretty badly.... :( Can anyone help with a method that avoids the problem? Even with the frustration I admit to a wry smile that computers can't do some calculations correctly!precision = the number of places to the right of the decimal point. double round(double value,int precision){ const int adjustment=pow(10,precision); return floor(value*(adjustment)+0.5)/adjustment;}double round(double value,int precision ){ double remainder=fmod(value,precision); if (remainder>(0.5*precision))value=value-remainder+precision); else value=value-remainder; return value;} -Dai
March 2, 201115 yr There are many numbers that cannot be represented exactly in floating point. For example 0.1 becomes 0.10000000000000006 as a double. I think the general solution is to use either BCD or integer arithmetic. If the problem arises with comparisions then replace: if ( a == B)with if (fabs (a - B) < 0.000001) Gerry Howard
March 3, 201115 yr Author Commercial Member Thanks Gerry. Given the current setup, converting to integer is going to be easier than BCD.-Dai
Create an account or sign in to comment