Sign in to follow this  
Guest dswo

The FSX Survey is flawed

Recommended Posts

I believe that the results of the survey will not correctly mirror the feelings of those of us that have completed it.I find it impossible to correctly indicate the answers to the questions about both "...the experience " and the "...frame rates" when faced with the stark choice of either "Good" or "Poor".Had I been given the choice of "Satisfactory" or "Acceptable" I would have opted for that middle ground. Suerly there is a grey, undefined area between good and poor............Cliff

Share this post


Link to post
Share on other sites
Help AVSIM continue to serve you!
Please donate today!

I agree on that. Would be nice to have something like "satisfactory"

Share this post


Link to post
Share on other sites

RE: The FSX Survey is flawed Some folks just aren't happy with anything. Craig

Share this post


Link to post
Share on other sites

Actually I think the survey is perfect. If you are HONEST, it forces you to make a very macro type choice. If you nitpick each item with too many choices you dont get a real picture of what is going on. Personally after taking it and viewing the results, it appears very accurate to me. I might change one or two questions but remember this is a survey taken inside a tight community of EXPERT users. It does make a difference. We should be able to make these types of choices If your being honest.Hornit

Share this post


Link to post
Share on other sites

I think it would be more accurate to simply say it's "unscientific". Having studied statistics and been involved in survey and test design I can tell you it is very difficult to design any survey that eliminates bias. Therefore this survey (like any otehr informal poll) is interesting but not terribly useful in trying to extrapolate the results to any population greater than those who responded.

Share this post


Link to post
Share on other sites

>I think it would be more accurate to simply say it's>"unscientific". Having studied statistics and been involved in>survey and test design I can tell you it is very difficult to>design any survey that eliminates bias. Therefore this survey>(like any otehr informal poll) is interesting but not terribly>useful in trying to extrapolate the results to any population>greater than those who responded.The AVSIM survey may be "flawed," but with 3,000+ respondents I don't think it can be dismissed as "informal," either.

Share this post


Link to post
Share on other sites

The survey represents the views of those members of AVSIM who bothered to respond. It can't be extrapolated to give any indication of the general users of FSX. It's also informal because there's no way to ensure that it even fully represents the views of AVSIM members as a whole - it only represents the views of those who responded. A formal survey would ensure that the views of a fully representative cross-section wre obtained.BTW this isn't a critisism of AVSIM - it's merely pointing out the limitations of this type of self-selecting survey which is all that AVSIM can use.

Share this post


Link to post
Share on other sites

>It's also informal because there's no way to ensure that it>even fully represents the views of AVSIM members as a whole ->it only represents the views of those who responded. A formal>survey would ensure that the views of a fully representative>cross-section wre obtained.If the survey had only 100 responses, sure. But with more than 3,000? Most political polls, including the Gallup polls, are based on a sample only half that size, and that's to estimate something for the whole U.S.** Curious how that can work? Read this: http://media.gallup.com/PDF/FAQ/HowArePolls.pdf

Share this post


Link to post
Share on other sites

>Had I been given the choice of "Satisfactory" or "Acceptable">I would have opted for that middle ground. >>Suerly there is a grey, undefined area between good and>poor............Grey, undefined areas, and middle ground don't help anyone get a good overview of anything.Sometimes very black and white answers work better, as they force you to take a firm stance in your response.

Share this post


Link to post
Share on other sites

But, scientific poll sampling techniques are designed to eliminate, as much as can be, bias. Even though the sample is small, the sample population in a properly run poll is very carefully chosen to insure the validity of the findings. A poll such as we have here can in no way be construed as scientific and the results are only meaningful as they reflect the opinions of those AVSIM members who chose to respond. Extrapolating these results to be representative of the universe of FSX users would be a serious misuse of the data and will not yield anything close to a valid finding.Doug

Share this post


Link to post
Share on other sites

> the results are only meaningful as they reflect> the opinions of those AVSIM members who chose to respond.No, with a sample this big, the sample will be representative of the whole population (in this case, AVSIM users).>Extrapolating these results to be representative of the>universe of FSX users would be a serious misuse of the data>and will not yield anything close to a valid finding.Agreed, in principle. But statistics -- on this scale -- isn't an all-or-nothing game. The AVSIM population is a subset of the general population, not an alien species. Sampling AVSIM members won't tell you everything about the larger population, but it will give you a hint of where the wind's blowing.

Share this post


Link to post
Share on other sites

David,What you are arguing is that 3000+ self-selected respondents are a good sample. In the Gallop PDF, they have numerous examples where the number of respondents are large, but the results are still biased. This is a passage from the PDF."The first one thousand people streaming out of a Yankees game in the Bronx clearly aren't representative of all Americans. Now consider a group compiled by selecting 1,000 people coming out of a Major League Baseball game in every state in the continental United States -- 48,000 people! We now have a much larger group -- but we are still no closer to representing the views of all Americans than we were in the Bronx. We have a lot of baseball fans, but, depending on the circumstances, these 48,000 people may not even be a good representative sample of all baseball fans in the country -- much less all Americans, baseball fans or not."The sampling must be done randomly as stated in the Gallop PDF to obtain confidence in the poll results. Since the respondents chose themselves for this poll, it is not random and thus may have some bias. Note I used the word "may" in the previous sentence. The poll may actually represent the true opinions of everyone who visits AVSIM and has "flown" FSX, but we cannot know this scientifically becase of the self-selection of respondents. Sorry for the long post. My career is in statistics and I just couldn't help responding (note the bias as I'm a self-respondent). Thanks,Joshua

Share this post


Link to post
Share on other sites

Joshua, you're quite right about the random vs. self-selected sample. Frankly, I hadn't thought about that. Still, for such a small population (meaning here, "regular AVSIM visitors), 3,000 is a very large sample. Now I'm off to watch election returns!

Share this post


Link to post
Share on other sites

>>Had I been given the choice of "Satisfactory" or>"Acceptable">>I would have opted for that middle ground. >>>>Suerly there is a grey, undefined area between good and>>poor............>>Grey, undefined areas, and middle ground don't help anyone get>a good overview of anything.>>Sometimes very black and white answers work better, as they>force you to take a firm stance in your response.>I have to disagree. Not everything is YES or NO. Example, Question 8:---------------------------* 8. When you have had issues with FSX, have you been able to get solutions? YesNo----------------------------If I answer YES or NO, there is an implication that I HAD issues. How do you answer the question if you did NOT have issues. To follow with question 9 regarding where I recieved help, once again this would imply that I had issues in the first place.In conclusion, most surveys remind me of the following:http://en.wikipedia.org/wiki/Lies,_damned_..._and_statisticsAs Richard Dawson would say: "Survey says"... always made me wonder who the heck they polled :)

Share this post


Link to post
Share on other sites

It is a "survey", not a statistical test. It has no predictive or inferential value whatsoever and, as a survey, doesn't purport any such. It does have a descriptive value, however. And this value provides more information about the current perception of FSX by the Avsim readers than a series of inflammatory or cranky posts.

Share this post


Link to post
Share on other sites

Professionally run polls take care to eliminate bias as far as possible. To do this they have to make some selection in their sampling. As an obvious example, the general population is about 50% men and 50% women. A poll in which 90% of the replies came from men would clearly be unrepresentative, so those conducting the poll ensure this doesn't happen. This applies to other differences, race, social class, age etc. Those conducting the survey on the ground will be given quotas of the different types of people to interview to avoid this bias.Self-selection introduces bias. It's just a fact of life that those who are very disappointed with anything (such as FSX) are, naturally, more likely to take the opportunity to express their feelings than those who are satisfied, or those who think it's "OK sort of".

Share this post


Link to post
Share on other sites

>Self-selection introduces bias. It's just a fact of life that>those who are very disappointed with anything (such as FSX)>are, naturally, more likely to take the opportunity to express>their feelings than those who are satisfied, or those who>think it's "OK sort of".This is reasonable. But when I look at the actual results, I notice a couple of things:1. About two thirds of respondents WOULD recommend FSX to a friend.2. This was the ratio a few days ago, when there were only 1,000 respondents, and it's still the ratio now that there are 3,000+ respondents.My inferences:a. It's not just, or even primarily, FSX-haters who are taking the survey.b. The initial sample (first 1,000) was predictive of AVSIM users who hadn't taken the survey yet (the next 2,000).

Share this post


Link to post
Share on other sites

What you say is of course true but it doesn't show that there's no bias. Suppose, to illustrate my point, that 10% are disatisfied and they all respond, but that only half of those are satisfied respond. Assuming (again to illustate my point) the total number of those who could respond to the survey is 5000 then there would be be 500 disatisfied responses, and 2250 satisfied responses ((5000 - 500) * 0.5). That gives an apparent ratio of 18% (500 / (2250 + 500)) who are disatisfied whereas the underlying figure is, by definition, only 10% (500 / 5000). In the case of the FSX survey, this suggests that there are more satisfied users that the survey apparently shows.

Share this post


Link to post
Share on other sites

>I believe that the results of the survey will not correctly>mirror the feelings of those of us that have completed it.>>I find it impossible to correctly indicate the answers to the>questions about both "...the experience " and the "...frame>rates" when faced with the stark choice of either "Good" or>"Poor".>>Had I been given the choice of "Satisfactory" or "Acceptable">I would have opted for that middle ground. >>Suerly there is a grey, undefined area between good and>poor............>>CliffNo, you had the choice of Excellant, very good, good, poor, very poor, and unusable. I would say anything in the poor and below would be considered unacceptable, unsatisfactry or unusable, and anything in the good to excellant range would be considered acceptable, satisfactory or excellant. Not sure where is the disconnect here, other than terminolgy.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this