Lies, Damn Lies, and Statistics

[PhilTaylor 45]

Unfortunately the XP-haters thread is locked, because I wanted to present some statistical analysis on "facts" about FSX and crashes presented by GoranM in that thread, in the interest of fair play.It was quoted in that thread that there are 30 pages of topics in the FSX CTD forum, as if that is somehow an indictment of FSX.It is true, that is the count of pages. That is all that is true about that number, however.As usual with statistics, a truthful comparison lies buried and needs analysis. Which I will deliver.If we are to use number of crash posts as a proxy for stability ( which was the original intent even though it is flawed ), one must look at relative numbers to weight the occurrences, to gauge a sense of how prevalent crashes are in one program versus the other, and once one has done that one must normalize by time duration so that we are comparing to a similar amount of time.And once that relative weighting calculation and a normalization is performed, an entirely different point of view comes forth.Specifically:

The FSX forums start in Oct 2006. The XP forums start in Dec 2007/Jan 2008. So a time duration normalization must be used, otherwise we are comparing apples to oranges.

And the frequency of CTD forum topics must be compared to the overall frequency of FSX forum topics, to gauge a percentage. And the XP forum must be examined to determine total number of crash threads as a percentage of total threads. That gives us a relative weight in both cases, so again we are comparing apples to oranges and not just using a single value in isolation to determine fact.

Note, my analysis assumes the forum moderators are diligent about moving posts on crashes from the FSX forum to the FSX-CTD forum, but I feel that is a safe assumption.FSX------There are 30 pages of CTD posts, and 3609 pages of FSX posts. 30/3609 is .0083, to 4 decimal places.When I normalize that to CTD posts that start Dec 2007/Jan 2008, I get 22 pages of posts in the CTD forum, 22/3609 is .0061.that is 8/10ths of 1 percent un-normalized, and 6/10ths of 1 percent normalized.in both cases that is less than 1%, as a ratio of crash posts to "usage" posts.XP---there is no separate forum, so it is necessary to scan the XP forum and determine crash posts. and then compare that to "usage" posts.my scan of the forum showed 10 crash posts out of 22 pages, with a distribution as follows ( an x means no crash post on that page ):

page crash posts

1 crash on exit (1)2 x3 x4 crash on exit (2), crash on calibration (3)5 x6 x7 x8 x9 crash on startup (4), settings crash (5)10 crash (6)11 x12 x13 x14 x15 graphic card crash(7)16 x17 x18 x19 crash at startup (8), crash (9), driver crash (10)20 x21 x22 x

with 20 posts per page, 10 is one half of a page. so that is .5/22, and the ratio is .0227, which is 2.3%.Comparitive Ratios---------------------------using the un-normalized values, I get .0227/.0083 = 2.73x, or XP forums show 2.73 times the crash posts of the FSX ratio. Thats 273% more likely to crash.using the normalized values, I get .0227/.0061=3.72x, or the XP forums show 3.72 times the crash posts of the FSX ratio. Thats 372% more likely to crash.Even dismissing 20% of the XP threads ( 2 out of 10 ) we get .4/22= .0182, which is still 2.18 or 218% more likely, as compared against the un-normalized value and 2.98 or 298% more likely than the normalized value.Moral of the Story-------------------------Be careful of quick use of numbers without statistical analysis, it might lead to an unexpected result. I won't use this statistical analysis to bash XP, I don't see that as a fruitful goal. I instead use it to point out the uselessness of quoting numbers like that.I would add a hope that the XP advocates would stop coming to MS forums and bashing Flight or Flight Sim. Or, at a minimum, do it with actual thought and analysis instead.Finally, I find both of these numbers to be in an 'acceptable' range for consumer entertainment software and neither number in and of itself 'proves' anything about either program, much less that one is 'better' than the other. Consumer preference, fickle as it is, to me is the only meaningful determination of 'better' since that is the measure by which we can see who spent their hard earned shekels and where.

[GoranM 3,762]

Far too many variables are missing from your equation.I admire the effort, though.

[bonchie 2,305]

His point went completely over your head in this case. Read the entire post.

[GoranM 3,762]

No, I got his point. He didn't get mine.All I said was there is a dedicated FSX CTD forum with 30 pages of threads. And I'll be honest, I find that amusing and surprising. (I've been the victim of many OOM's and CTD's myself) Not to mention what is arguably the best developer for FSX recommending a 64 bit OS because of FSX's memory limitations. Yet the mere mention of that, and other FSX users interpreting that as a negative, brings out the defensive in quite a few people.These are the facts. No speculation. No assumption.That was all I said. Nothing more and nothing less. Draw from that what you will.

[PhilTaylor 45]

No, I got his point. He didn't get mine. All I said was there is a dedicated FSX CTD forum with 30 pages of threads. And I'll be honest, I find that amusing and surprising. (I've been the victim of many OOM's and CTD's myself) Not to mention what is arguably the best developer for FSX recommending a 64 bit OS because of FSX's memory limitations. Yet the mere mention of that, and other FSX users interpreting that as a negative, brings out the defensive in quite a few people. These are the facts. No speculation. No assumption. That was all I said. Nothing more and nothing less. Draw from that what you will.

as I saId, a single number is meaningless. when taken in context, it has meaning.you can keep trying the single number bit, but any one who is thoughtful will now see it for what it is. randomness.

[GoranM 3,762]

as I saId, a single number is meaningless. when taken in context, it has meaning.you can keep trying the single number bit, but any one who is thoughtful will now see it for what it is. randomness.

I don't have to "try" anything. It's there. And it's not because of a specific hardware problem. I used to fix computers for a living so I know my way around one, and I still got CTD's and OOM's. As did many others. And I hope you're not suggesting I am not "thoughtful". I see the figures and I see the context. But sometimes, a single number is all that is needed.

[PhilTaylor 45]

I don't have to "try" anything. It's there. And it's not because of a specific hardware problem. I used to fix computers for a living so I know my way around one, and I still got CTD's and OOM's. As did many others. And I hope you're not suggesting I am not "thoughtful". I see the figures and I see the context. But sometimes, a single number is all that is needed.

I am stating, factually, the number doesnt mean what you state it does. And have presented the analysis to back it up.Continuing to quote the single number will prove you are not being "thoughtful" as contextually it does not mean what you are trying to say it does.

[GoranM 3,762]

I am stating, factually, the number doesnt mean what you state it does. And have presented the analysis to back it up.Continuing to quote the single number will prove you are not being "thoughtful" as contextually it does not mean what you are trying to say it does.

Ok, Phil.Having a dedicated CTD forum means nothing...when taken into the context you have presented.Can we be happy now?

[Jacoba 55]

as I saId, a single number is meaningless. when taken in context, it has meaning.you can keep trying the single number bit, but any one who is thoughtful will now see it for what it is. randomness.

And you can give it any meaning you want... take this from a guy who spends his life neck deep in finance...If the mortgage meltdown of 2008 proved anything, it proved that anything can be painted in any context you want... and with enough brush strokes even the most savvy of folks will buy it... literally

I am stating, factually, the number doesnt mean what you state it does. And have presented the analysis to back it up.Continuing to quote the single number will prove you are not being "thoughtful" as contextually it does not mean what you are trying to say it does.

I can present a pile of analytical studies done by some of the greatest financial minds on the planet and presented to the financial world in the summer of 2008, just weeks before banks collapsed and MBS's blew up the economy... that all said Lehman Brothers and Bear Stearns were rock solid institutions and MBS (Mortgage Backed Securities) were a sound investment too...But we now know that all of those points were total bunk...

[WarpD 824]

Actually, Phil's analysis is spot on (for it's purpose).Goran's statements are neither empirical nor statistically correct because they're completely subjective. Statistical analysis is never subjective. That alone invalidates his point which is: attempting to prove FSX is more unstable than X-Plane. There are no actual facts to support that claim.Let's toss in the sheer number of sales. Is it safe to say X-Plane has sold as many copies as FSX has? I suspect not. Based on that alone... the 'sample' is skewed in X-Plane's favor.I could go on and on and on... but the bottom line is: Goran's point is completely and totally invalid. Which was the point of Phil's post.

[GoranM 3,762]

Actually, Phil's analysis is spot on (for it's purpose).Goran's statements are neither empirical nor statistically correct because they're completely subjective. Statistical analysis is never subjective. That alone invalidates his point which is: attempting to prove FSX is more unstable than X-Plane. There are no actual facts to support that claim.Let's toss in the sheer number of sales. Is it safe to say X-Plane has sold as many copies as FSX has? I suspect not. Based on that alone... the 'sample' is skewed in X-Plane's favor.I could go on and on and on... but the bottom line is: Goran's point is completely and totally invalid. Which was the point of Phil's post.

Defensive.All I said was there are 30 threads in a dedicated FSX CTD forum and I found that funny.Draw from that what you will.I'm sorry to disappoint you but that is all that was said...no matter how much of a twist you want to put on it.

[HughesMDflyer4 470]

Defensive.All I said was there are 30 threads in a dedicated FSX CTD forum and I found that funny.Draw from that what you will.I'm sorry to disappoint you but that is all that was said...no matter how much of a twist you want to put on it.

Now he's in denial.

[GoranM 3,762]

Now he's in denial.

LMFAOOk? Brandon. You're a funny guy!

[ahuimanu 50]

Would you two please consider taking this over to Hangar chat? The establishment of fact notwithstanding, this sort of exchange just stinks the place up.Thank you.

[Guest jahman]

There is of course also the qualitative question of whether an OOM due to extending FSX waaay beyond its intended design (e.g. with 4096 textures and huge LOD radius with autogen-dense scenery) can even be called a bug. But I digress...And I'll digress s'more: Perhaps the reason we FSXers are so sensitive to CTDs is we've spent money on add-ons and sim hardware as we've never done before with all versions of FS combined. Having FSX run with that much excellent scenery and sofisticated aircraft means the bar has been raised much higher thus a CTD hurts so much more. In this sense FSX is a victim of it's own success (as well as five years without a sequel).Just my $0.02.Cheers,- jahman.