Sign in to follow this  
Guest bartels

5700 - how to extract 7 in xml (rounding number problem)?

Recommended Posts

HiI've really pulling my hairs on this one. I guess it's the integer problem striking me again at full strenght, but this time I'm not able to find a workaround. The number 5700 is on stack, how do I extract the 7 and have it be 7? I've tried all sorts of %, div, flr, ceil, near, negating (!), sqr d * and so forth, but I'm only getting the number 8 out. 5600 works ok though.I've tried using if and els, but (obviously) I'm not getting the hit where I want it. The div command, which is supposed to do integer division, does it work as expected?If this is a result of binary numbers, I'm sure some experts here have found a neat workaround for this?Oh, btw, the int (same as flr on positive numbers?) command is not mentioned in the SDK, yet Microsoft themselves use it a number of times. Heeelp.....

Share this post


Link to post
Share on other sites
Help AVSIM continue to serve you!
Please donate today!

The problem might be taht internally floats are used and using integer operations on it can lead to problems, e.g. an int operation cuts the digits, it does not round! To round try adding 0.5 and then use ceil to make proper roundings from floats to ints.Arne Bartels

Share this post


Link to post
Share on other sites

Add a small number before flooring, huh? Neat! :) Can't belieave I didn't see that one. Hopefully it will work. Not sure if integer math on float is the actual problem, although this time Excel passed the test correctly. That was not the case earlier, where floats produced rounding errors on extremely simple maths (due binary representation of floats).Thanks.

Share this post


Link to post
Share on other sites

Not an arbitray small number, exactly 0.5. This just shifts the truncating problem in a region where it is safe. It is 0.5 to make 0.5

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this