March 28, 200422 yr HiI've really pulling my hairs on this one. I guess it's the integer problem striking me again at full strenght, but this time I'm not able to find a workaround. The number 5700 is on stack, how do I extract the 7 and have it be 7? I've tried all sorts of %, div, flr, ceil, near, negating (!), sqr d * and so forth, but I'm only getting the number 8 out. 5600 works ok though.I've tried using if and els, but (obviously) I'm not getting the hit where I want it. The div command, which is supposed to do integer division, does it work as expected?If this is a result of binary numbers, I'm sure some experts here have found a neat workaround for this?Oh, btw, the int (same as flr on positive numbers?) command is not mentioned in the SDK, yet Microsoft themselves use it a number of times. Heeelp.....
March 28, 200422 yr The problem might be taht internally floats are used and using integer operations on it can lead to problems, e.g. an int operation cuts the digits, it does not round! To round try adding 0.5 and then use ceil to make proper roundings from floats to ints.Arne Bartels
March 29, 200422 yr Add a small number before flooring, huh? Neat! :) Can't belieave I didn't see that one. Hopefully it will work. Not sure if integer math on float is the actual problem, although this time Excel passed the test correctly. That was not the case earlier, where floats produced rounding errors on extremely simple maths (due binary representation of floats).Thanks.
March 29, 200422 yr Not an arbitray small number, exactly 0.5. This just shifts the truncating problem in a region where it is safe. It is 0.5 to make 0.5
Create an account or sign in to comment