With a lot of discussion within many review comments, there are pros and cons of "rating" a product based on one reviewer's opinion and the subsequent "review" of the report by the editor.When I started with AVSIM in 2000, we had a 1 to 5 star rating system, where 1 denoted a poor product, a 3 denoted a product met theexpectations of the day, and a 5 meant that the product broke new ground and raised the bar for the next developer to meet or exceed.As technology (and MSFS) marched on, 4.5 and 5.0 were becoming the norm as the latest release exceeded the last. So we did away with the 1-5 rating and introduced the "Gold Star Award" reserved for those products that were now "ground breaking" in the eye of the reviewer. I, as the editor, can't fly all of the products that are submitted for review, although I wish I could and still pay the mortgage. But as we are a volunteer staff, we have to rely on you, the readers, to step up and take on a role as a reviewer to give back to your fellow simmers by offering an unbiased, best of our ability review of the product we are assigned to review. We have reviewers male and female, aged 16 to 60+, all with unique writing skills and experiences in the field of aviation, be it real world or simulated.So back to the subject at hand. I'm looking for your input as to what you would like to see in a rating system, if any? In the end, this is a website for you, the flight simmer. Just because I think this is the best product since the Wright Flyer, and it runs like a charm on my water cooled, 6-core i7 with 12MB RAM and multiple solid state HDs (yes I got my PC from Jetline Systems as advertised on our Front Page) doesn't mean you will get the same results on your slightly older PC. So do you want us to rate it with 1-5 stars or 0-100 or stay with the Gold Star or do away with the rating system altogether? It was mentioned in one review that perhaps we could let you, the reader, give your honest opinion of your experiences with the product and suggest a rating to be published shortly after the review has gone live.....but wait...you're already doing that through the forums anyway when you respond to a review you've read. I'm open to suggestions and sometimes change is for the better.

Thank you for starting this thread. I have been thinking to myself lately that maybe it's time to rethink the rating system and was considering starting a discussion, but it's so much better coming from your end. :Applause: There is obviously never going to be a perfect system nor are we ever going to have total agreement. But how about this as a basis for discussion?I suggest a kind of "triage" approach. Three ratings...1 = below par. The product does not meet acceptable standards for current payware2 = meets all expectations. A product that meets reasonable and acceptable standards for current payware3 = exceeds expectations. A product that really stands out as outstanding.Gold Star - only awarded to a product that has recieved a 3 rating, has had a positive reaction in the forums and is tested by two or three independent people with good standing in the community - a vague concept I know but I'm sure a list could be made of volunteers who have been around for a while and are knowledgable. They would not be writing a review - merely confirming what the reviewer and the general forum discussions claim.The 1 - 2 -3 rating would not be very discriminate but it also eliminates a lot of the "shades of gray" that the old system had. What really makes the difference between a 4 and a 4.5? Impossible to define of course. But below par, par and above par are pretty clear categories.One other thing I recommend. Instead of putting the rating in a seperate box or at the top of the review, I would "bury" it within the text of the review so people really have to read the review rather than just seeing a "2" and concluding that it's not all that great. The review should clearly state the criteria that led to it being considered a 1 or a 2 or a 3 before the rating is "revealed." That way a 2 would not be the proverbial "kiss of death" - people would read the review, see that it is a product that is definitely worth buying even if it isn't "outstanding."Whatever system may be adopted, I do not recommend that we reinstate the old criteria for a 5 that it had to demonstrate some kind of innovation. A product that is outstanding in overall quality might not be innovative but if it makes full use of the "state of the art" techniques, then it deserves to be recognized as excellent. I remember agonizing over this when I was an AVSIM reviewer back then - a wonderful product that I wanted to be recognized as outstanding but didn't "innovate" so I couldn't in good conscience give it a 5 - even though in some cases it deserved it.I hope we can come up with something that gets generally accepted as I do think some improvement is neededIanps. Whatever you decide, I hope you can eliminate some of the incredibly arbitrary criteria that are sometimes used. One recent review criticized an outstanding add-on scenery because it didn't have "bird sounds." I have over 400 add-ons and none of them have "bird sounds" nor have I ever considered they should. :Thinking:

As for me I liked the 1 to 5 star rating system you used to have. It was clear and easy to understand.

Hi Robert,I always liked the old 1-5 rating system, but I remember it generating a lot of complaints because each reviewer is going to look at a product a little differently. What one reviewer considered a three might have been a four in another reviewer's book. What I always thought was missing was a thorough summary of WHY the number was given. For example, if someone gave a product four stars, I would have liked a paragraph or two explaining why it didn't deserve five, and what made it better than a three. I also thought that a 1-10 star rating would have provided a bit more room to be more specific. In a 1-5 star system, each level is 20% higher than the one before it, which is a big jump. One of the smartphone review sites I frequent uses a 1-100 rating system, and often breaks down the numbers into halves, such as 94.5. This gives them a LOT more room to be specific about why the rating is what it is. I don't think AVSIM needs to go quite that far, but a 1-10 would be nice. Some products in a 1-5 rating might be on the high or low end of the star rating given, which can be remedied by having more numbers to choose from. For example, if a product earns four stars, it would get about an 8 in a 1-10 system. But if it just barely made four stars, it can get a 7 instead. Or if it is a strong four star product, almost a five, then maybe it can get a 9. Or, of course, you could break it down even further and have 70-90 to choose from for a four star product in a 1-100 rating system.

Good feedback guys. I hope we can get some more members to get into the dialog and contribute some ideas. So far it seems we are agreed that the old rating system had its merits, as well as drawbacks. I remember vividly when it was discontinued and why - but sadly that meant the trolls won. There were legitimate reasons to challenge some of the ratings that were given, as discussed on here. But I think it could be fixed. As for the habitual whiners - they are going to whine no matter what system is used...

to answer the question directly - To star or not to star? - Yes, we need to award AVSIM Stars to those deserving products so1. Our readers/members will know how the reviewer feels about the product deserving of such.2. The Developers will hopefully strive to make good enough products to receive those stars.Whether we continue to use the current ONE star system or a 5-star or even a 10-star is not as important to me as the uniform use of evaluation criteria to award those stars.I really don't have any problem with the current system. I would like to see more text as to why or why not a product did or didn't receive the gold star provided someone thought it worthy.I just finished reading one of the most recent reviews that I feel deserved the AVSIM Gold Star, which it did, but there was not one word attributed specifically to that award. Just a gold star at the top of the review. The reviewer stated it should be obvious based on all the good things he had to say about the product throughout the review. Nothing is obvious unless we make it obvious.I think a product can even have a few blemishes and still receive a AVSIM Gold Star, provided it is head and shoulders above the competition. If there is no competition or limited competition then we may have to look at it a little closer.I guess as long as those items receiving the Gold Star always float to the top of the conversation and expectations, we are probably doing it right. Those that get stuck floating to the top are probably suspect. But, I don't think the reputation of a developer is enough to get a gold star for everything produced. There may be exceptions to that, maybe PMDG, maybe a couple of others. I think we have to look at the total package: What it does, how well it does it, does it have sufficient manuals, is it priced fairly, how are updates handled, how is support handled, does it have paintkit if needed, does it have a tutorial if one is normally expected. etc.Usually the reviews have a closing or summary that pulls it all together with maybe Likes and Dislikes. I think that should be standardized for all reviews. If a product is to receive the coveted AVSIM gold star this is where is should be justified in writing. I think it important to know a little about the reviewer, his or her system used for the testing, and some remarks as to the quality, fidelity, ease of use, efficiency of the product vs its intended use.I sure wish our forum had a spell checker. Maybe the new March 5 update will have one.Ray

I've always found a like and dislikes list to be a much more useful summary of a review (and not just flightsim addons) than an arbitrary number (or stars).

I agree with everything you have both said. What I would hope for is a rating system that is designed to be as objective as possible so the stars are never awarded "arbitrarily." But as of now, the system is "all or nothing" and there is no clear criteria, at least for us readers, as to how one excellent product gets a star and another that also got a flawless review doesn't

I agree with everything you have both said. What I would hope for is a rating system that is designed to be as objective as possible so the stars are never awarded "arbitrarily." But as of now, the system is "all or nothing" and there is no clear criteria, at least for us readers, as to how one excellent product gets a star and another that also got a flawless review doesn't

Maybe we should all contribute our thoughts as where to draw the line and let someone or some small committee draft the criteria then make it a requirement for all reviewers.Ray

Maybe we should all contribute our thoughts as where to draw the line and let someone or some small committee draft the criteria then make it a requirement for all reviewers.Ray

I just sent a PM to the Review administrator who started this thread. As long as I know we won't be just talking but can maybe come up with something that would be considered for adoption, I'm in! Hopefully he will post soon and give us some guidance as to how he sees this topic functioning. I get the feeling from the original post that the intention was to come up with some real suggestions and implement a new system or at least tweek the existing one.

Yes, I think a 0 to 5 star system would be good.Vic

Yes, I think a 0 to 5 star system would be good.Vic

I'm fine with it as along as there are clear criteria and the standards for evaluation are enforced. I still prefer the "triage" approach - there's less room for ambiguity. It's either below par (with full explanation why), it meets the bar of current payware (with full explanation) or it exceeds the bar (full explanation AND independent peer testing of the product before publication). Maybe a .5 option when it is a tough call but only with full explanation of why it "fell in between." I would not use stars except for the top products which should continue to get the current AVSIM star. I would just give a numeric score at the conclusion of the review so people won't just see stars at the top and get an instant bias for or against the product unless it's total garbage and then maybe give it a lemon award at the top. (Also with prior independent peer review)I would also like to see a seperate library list with a link for all top products as well as listing them within the existing categories. I think it would be interesting and useful to see at a glance all products that earned 5 stars under the old system and one star under the new system

I'm leaning towards going back to the 5 star system, with a Gold Star for the top products. I'd also like to see an actual numeric score from the reviewer, with justification as to why the product scored what it did.I'll review your comments and our system with the executive at our next meeting.

I'm leaning towards going back to the 5 star system, with a Gold Star for the top products. I'd also like to see an actual numeric score from the reviewer, with justification as to why the product scored what it did.I'll review your comments and our system with the executive at our next meeting.

Sounds great Robert. I think you will find a willing group to work with you if you want feedback and discussion to refine it. Count me inIan

I like the idea of going back to the 1-5 star rating. What I have seen in other forums that rate video games is a way for the general users to cast a vote 1-5 as well. So what you end up with is the Avsim Review Rating and then a user review rating along side it. (You can go to the IGN game review website to get a clear picture of what I'm talking about) I've always found that particularly useful since you get a base review conducted by a staff reviewer + a general users rating to measure it against. So that if one of your staff reviewers rated a product at 5 stars and the average rating of 25 users was 3 stars then you can sort of assume that perhaps the official review was a little more positive than most people think. Just and idea.