AVSIM Review feature

AVSIM Editorial AVSIM Review Philosophy By Maury Pratt, Managing Editor

We've had considerable feedback from our forums and the "Feedback" feature in our reviews questioning our philosophy—and motivation—both as to why we select certain products for review, and the criteria our reviewers apply in rating them. We think our readers deserve an answer. As you know we run reviews of freeware, shareware and commercial products. Our reviewers seek out as many commercial products for review as they can find time to cover, knowing that you want as much information as possible before making a purchasing decision. As for shareware, and especially freeware offerings, we review those as well because they often serve to highlight advances in development technique and to demonstrate effective application of new development tools. We realize that, as some have said, you can download these without risk and decide for yourself—nevertheless our freeware reviews do much to chronicle advances in the art (literally). That's why we run our "Bravo Zulu" Awards year-end feature—to showcase technological advances and to acknowledge outstanding work. We award the "AVSIM Gold Medal for Excellence in Flightsim Design" to recognize and encourage innovation, whether freeware or commercial. And frankly we think the inevitable tension between those of freeware and commercial persuasion is healthy. Freeware puts pressure on commercial developers and publishers, and that's to everyone's benefit. Not just for AVSIM's readership, but for the legions who choose to buy off-the-shelf in software stores. On the question of "fairness" We've struggled with the question as to whether we should apply the same criteria in reviewing freeware offerings that we use for commercial products. We've come down on the side that all products—whatever their source—should be evaluated against a common, contemporary standard of excellence. Even so, we're cautious about singling out freeware for public criticism unless we feel we're being constructive in doing so. We're keenly aware that freeware authors are motivated not by the prospect of financial return but by their love for the hobby and the desire to contribute work of value to the entire community. That's what has made Microsoft's Flight Simulator series, and now TRI's Fly! so rich and so absorbing for all. We certainly have no desire to discourage anyone from offering his or her efforts for all to enjoy. So we'll probably pass on reviewing products that still "need work." But not always—a reviewer occasionally selects an offering for review because the author (or group) has a public following and hence feels that product should be examined, for better or worse. Where our reviewer finds fault—even in a freeware offering—we believe that comparison with similar products is in our reader's interests. And hopefully it's a spur to that author to take the criticism to heart in honing his skills. But we were wrong in posting a review recently of one of Project Freeware's earlier products, their B-737 series for FS98. That that review was not written by one of our regular review staff is beside the point; it's our call on what we publish. We removed that review because readers rightfully objected to the reviewer's implied comparison of a plane produced some three years ago, using development tools of the day, with today's products. But even here the outcome—as judged from the more reflective messages in this week's Project Freeware forum—proved to be productive as this review raised issues concerning flight model fidelity to real-world aircraft that may yet benefit future development. And that's our goal. Some suggest that we review only new or recently released products. For the most part that's what we do. Others, though, ask that we also continue to review FS98 freeware for those who have opted to stay with that simulator for the time being. So we'll continue to look for those add-ons that are in our opinion noteworthy. Who decides what to review? That call is essentially the reviewer's to make. Not to be "master of the obvious," researching and writing reviews is a time-consuming process. The time they are able to devote to reviewing is as limited as it is for freeware and shareware developers. As AVSIM Online enjoys a staff of reviewers sufficiently large to assure subject variety, we are comfortable with our policy to encourage individual reviewers to select items for review that particularly interest them—they need motivation just as do the authors who develop the products they review. As for reviewing commercial products, it's our managing editors' job to ensure that qualified reviewers are assigned to review newsworthy products. Sometimes the product is so comprehensive that we assign two or more to write a team review. How well does AVSIM Online manage the review process? We've said this before, but it bears repeating: we are resolute in keeping editorial content—including reviews—totally devoid of influence from Avsim's advertising side. Our record speaks for that, and our advertisers understand why this is essential to their well being as well. And I can assure everyone that all our reviewers feel more than strongly about adhering to this policy. We think the record shows that our reviewers are conscientious and fair minded. They are expected to be mindful that adverse findings may be inherent in the development tools used (and in FS or Fly! themselves)—and be identified as such. Further, we insist that adverse results that could result from an incomplete understanding of the product be verified with the author before publication. We're not infallible in this regard, but it's a point I insist on as Avsim's Review Editor. And our policies are by no means ad hoc. Each of our staff reviewers receive and accept formal review policies and authoring guidelines as a condition of serving on Avsim's staff. We have explicit review criteria, and insist on their providing a written analysis against that criteria with each review. That discipline adds to the burden on the reviewer's time, but we feel it necessary in assuring that we are as consistent as possible in assigning review ratings. All things considered, we believe our readers find AVSIM Online's reviews helpful, informative and interesting. We seek and accept constructive criticism, of course; that's part of our "rating" and is an important influence in improving our work and deserving your trust. So don't hesitate to tell us (constructively) how you think we're doing. Please use the review "feedback" feature, and of course we look for opinion and guidance in the forums. We're doing our best to make AVSIM Online your preferred source for everything about our fascinating hobby.
© 2000 - AVSIM Online All Rights Reserved

AVSIM Editorial

AVSIM Review Philosophy

By Maury Pratt, Managing Editor

We've had considerable feedback from our forums and the "Feedback" feature in our reviews questioning our philosophy—and motivation—both as to why we select certain products for review, and the criteria our reviewers apply in rating them. We think our readers deserve an answer.

As you know we run reviews of freeware, shareware and commercial products. Our reviewers seek out as many commercial products for review as they can find time to cover, knowing that you want as much information as possible before making a purchasing decision. As for shareware, and especially freeware offerings, we review those as well because they often serve to highlight advances in development technique and to demonstrate effective application of new development tools. We realize that, as some have said, you can download these without risk and decide for yourself—nevertheless our freeware reviews do much to chronicle advances in the art (literally).

That's why we run our "Bravo Zulu" Awards year-end feature—to showcase technological advances and to acknowledge outstanding work. We award the "AVSIM Gold Medal for Excellence in Flightsim Design" to recognize and encourage innovation, whether freeware or commercial. And frankly we think the inevitable tension between those of freeware and commercial persuasion is healthy. Freeware puts pressure on commercial developers and publishers, and that's to everyone's benefit. Not just for AVSIM's readership, but for the legions who choose to buy off-the-shelf in software stores.

On the question of "fairness"

We've struggled with the question as to whether we should apply the same criteria in reviewing freeware offerings that we use for commercial products. We've come down on the side that all products—whatever their source—should be evaluated against a common, contemporary standard of excellence. Even so, we're cautious about singling out freeware for public criticism unless we feel we're being constructive in doing so. We're keenly aware that freeware authors are motivated not by the prospect of financial return but by their love for the hobby and the desire to contribute work of value to the entire community. That's what has made Microsoft's Flight Simulator series, and now TRI's Fly! so rich and so absorbing for all.

We certainly have no desire to discourage anyone from offering his or her efforts for all to enjoy. So we'll probably pass on reviewing products that still "need work." But not always—a reviewer occasionally selects an offering for review because the author (or group) has a public following and hence feels that product should be examined, for better or worse. Where our reviewer finds fault—even in a freeware offering—we believe that comparison with similar products is in our reader's interests. And hopefully it's a spur to that author to take the criticism to heart in honing his skills.

But we were wrong in posting a review recently of one of Project Freeware's earlier products, their B-737 series for FS98. That that review was not written by one of our regular review staff is beside the point; it's our call on what we publish. We removed that review because readers rightfully objected to the reviewer's implied comparison of a plane produced some three years ago, using development tools of the day, with today's products. But even here the outcome—as judged from the more reflective messages in this week's Project Freeware forum—proved to be productive as this review raised issues concerning flight model fidelity to real-world aircraft that may yet benefit future development. And that's our goal.

Some suggest that we review only new or recently released products. For the most part that's what we do. Others, though, ask that we also continue to review FS98 freeware for those who have opted to stay with that simulator for the time being. So we'll continue to look for those add-ons that are in our opinion noteworthy.

Who decides what to review?

That call is essentially the reviewer's to make. Not to be "master of the obvious," researching and writing reviews is a time-consuming process. The time they are able to devote to reviewing is as limited as it is for freeware and shareware developers. As AVSIM Online enjoys a staff of reviewers sufficiently large to assure subject variety, we are comfortable with our policy to encourage individual reviewers to select items for review that particularly interest them—they need motivation just as do the authors who develop the products they review.

As for reviewing commercial products, it's our managing editors' job to ensure that qualified reviewers are assigned to review newsworthy products. Sometimes the product is so comprehensive that we assign two or more to write a team review.

How well does AVSIM Online manage the review process?

We've said this before, but it bears repeating: we are resolute in keeping editorial content—including reviews—totally devoid of influence from Avsim's advertising side. Our record speaks for that, and our advertisers understand why this is essential to their well being as well. And I can assure everyone that all our reviewers feel more than strongly about adhering to this policy.

We think the record shows that our reviewers are conscientious and fair minded. They are expected to be mindful that adverse findings may be inherent in the development tools used (and in FS or Fly! themselves)—and be identified as such. Further, we insist that adverse results that could result from an incomplete understanding of the product be verified with the author before publication. We're not infallible in this regard, but it's a point I insist on as Avsim's Review Editor.

And our policies are by no means ad hoc. Each of our staff reviewers receive and accept formal review policies and authoring guidelines as a condition of serving on Avsim's staff. We have explicit review criteria, and insist on their providing a written analysis against that criteria with each review. That discipline adds to the burden on the reviewer's time, but we feel it necessary in assuring that we are as consistent as possible in assigning review ratings.

All things considered, we believe our readers find AVSIM Online's reviews helpful, informative and interesting. We seek and accept constructive criticism, of course; that's part of our "rating" and is an important influence in improving our work and deserving your trust. So don't hesitate to tell us (constructively) how you think we're doing. Please use the review "feedback" feature, and of course we look for opinion and guidance in the forums. We're doing our best to make AVSIM Online your preferred source for everything about our fascinating hobby.