Avsim down time

[TechguyMaxC 516]

I would like to give the site administrator(s) my feedback about the frequency and length of down time for this site.  I intend the following to be inquisitive/informative in nature, and not a criticism of any person or organization.  

 

I visit approximately 20 web sites per day, several of which are forums.  None of those sites experience the number of down time incidents as Avsim.  Also, when the site does go down, it seems to remain down for up to several hours.  This seems to have gotten worse in the past month or so, but as a visitor to Avsim since the late 90's (when it played host to the MSFS Benchmark sub-site and forum), I can say that the number of "outages" over that time is far more than any other site I have visited so it is not merely a recent phenomenon.  

 

I am curious as to the specific cause of this issue.  Is there too much traffic for the server(s)?  Are the servers owned by Avsim or provided by a web host?  Are they co-located or do you own the building in which they are located?  What are the specs of the servers and how many are there?  How much bandwidth do you have access to?  Peak simultaneous user count?  Average users per 24/hours?  I believe with answers to these questions (and a few others) the problem(s) can be solved.  If you need advice relating to technology, I am an IT professional and would be more than happy to offer my services to consult on the matter at no charge.  If you would like to speak with the owners/administrators of other communities that I am involved in, I can put you in touch with them as well.  

[ckyliu 1,749]

Avsim forums have been more reliable (for me) since they made upgrades to their hard disk drives a few weeks ago... until today. Today it's been intermittent service since about 1030 GMT up until about 1530 GMT, with a variety of "Unable" and 502 errors.

 

Previous to those HDD upgrades Avsim went down every day with 503 errors from approx 0830 GMT to 1000 GMT (this has been covered elsewhere in great depth so I'm not going to detail that any further, suffice to say it seems fixed now)

 

Ultimately though, I have to agree with the OP; Avsim has by far the worst uptime of any website I've regularly visited over the last few years. I do appreciate this is a free-to-use service run by volunteers, but the facts regrading uptime remain.

[TechguyMaxC 516]

This user brings up another perspective with an interesting point:

 

http://www.avsim.com/topic/474475-pmdg-customer-forum-deserve-a-better-forum-platform/#entry3293111

[zmak 331]

If its a money issue then perhaps have monthly targets with a live counter going, it seems to work for a lot of websites. Tapa talk would be kind of nice to have as well

[el_kab0ng 29]

I've personally never seen a 'donation' site with a board of directors, either. With that sort of formality, there should be a much more reliable environment than is present today.

[Jimm 437]

Personally, if every single member gave $1 a month as a subscription fee, that would more than pay for enough people to maintain this site and less issue with downtime.  I would be more than happy to pay that.

 

-Jim

[TechguyMaxC 516]

Personally, if every single member gave $1 a month as a subscription fee, that would more than pay for enough people to maintain this site and less issue with downtime.  I would be more than happy to pay that.

 

-Jim

 

I'm sure lots of people would be happy to contribute to the site.  Let's determine the problem before we throw money at it though.   :wink:

[Jimm 437]

I'm sure lots of people would be happy to contribute to the site.  Let's determine the problem before we throw money at it though.   :wink:

Isn't that what the board of directors is for?

 

-Jim

[Chase 303]

Thank you everyone for expressing your concerns.  Please keep in mind the following (and yeah by now you're saying we've heard this before, I know):

 

• We are a donation based website with some supplemental income from FS advertisements
• We are staffed entirely by volunteers
• Our IT services are provided pro bono by an IT guy who has a demanding full time job
• Each of us has a full time job outside of AVSIM and devote a lot of our free time to the site away from our families

We are on a path to bringing the site where it needs to be.  Why do we have seemingly constant issues lately?  The honest and most humble truth is, we don't have thousands of dollars laying around to pay someone full time to do the work.  Also, we are working with aging equipment that is still useable for the foreseeable future but requires some "refreshing" and tweaking.  Basically, we are doing load balancing between two servers, and it's something AVSIM has never done before.  Sure, we used to have our library and forum on two separate servers, but we have true load balancing happening now where both servers are sharing loads and talking to each other to serve one product (also add a very new and shiny SAN device in there between the two).  Because of this change, there are some hiccups here and there.  Small ones that get fixed as soon as we notice/hear about them, and as soon as we can get our volunteer IT guy in to fix it.  We have a professional grade IT equipment, with part time IT help.  It's help we are most grateful for, but is limited.  You all remember what happened the last time someone offered us IT help when we needed it most -- we were nearly destroyed.  I don't call it paranoia, but we do call it extreme caution when we have hand selected an IT guy that's been with us for a long time.

 

We do bits and pieces of work as we can, little by little.  Our IT guy has spent several of his entire weekends (including a holiday weekend) at the server location doing diagnostics, installing new hardware, etc.  On his time off he's working on the software side of things trying to get things where they need to be.  This has been a rather expensive endeavor (already exceeding the entire year's donations), and we appreciate your support and understanding.

 

Some of these whacky conspiracy threads around the internet might lead you to believe we are just raking in the dough, it's the farthest thing from the truth.  We barely make ends meet each month to pay the bandwidth expenses alone.  Tom built AVSIM has a place where we would be the voice of the community without being a commercial "for profit" website.  I don't see us ever changing that, because that's not what Tom's idea was about.  

 

Regarding the what I consider slightly cheeky but also kind of "punch in the gut" comments about the board of directors... Please remember this title does not mean we make money... No one makes money at AVSIM and never has (please see paragraph above).  There were some months when Tom would pay for things out of his own pocket.  I know for a fact Tom ended up footing a good chunk of the expenses of our last convention just because the show must go on and he always put the community first--that was the just the kind of guy he was.  He would never want anybody to know it though.  The board of directors was a group of people Tom hand picked to keep his legacy alive and has the responsibilities of overseeing the operations and decisions since his passing.  

 

I speak as a volunteer of AVSIM today, a volunteer that loves this community and loves this website.  I have been a volunteer for AVSIM since I was 14.  In many ways, AVSIM is a part of who I am because I've known it since almost as long as I can remember.  So please, know that the times are not easy right now, but we are working forward to a better tomorrow.  Bit by bit, piece by piece, things are going to get better, as they already have... Performance wise, I think we are lightyears ahead of where we were only months ago (notice page loads aren't 12 seconds anymore).

 

Thank you, sincerely, again for all of your support... I do see very good things for AVSIM in the future with the trajectory we are on and the wonderful community we have supporting the efforts along the way.

 

Chase

[Jean-Jacques 120]

Hi Chasse,

 

A very professional response to the community members which should discourage further criticisism of volunteer work by a few members which benefit all of us. Regarding required operating funds, I would support a yearly fee proportional to the average yearly shortfall. Food for thought.

 

JJ

[zmak 331]

Hi Chasse,

 

A very professional response to the community members which should discourage further criticisism of volunteer work by a few members which benefit all of us. Regarding required operating funds, I would support a yearly fee proportional to the average yearly shortfall. Food for thought.

 

JJ

These offerings of support always go unanswered and Im sure they have their reasons for not going down this path.

Here is is good example of it in action, go to the top right for the counter  http://vattastic.com/

[TechguyMaxC 516]

Chase, thanks for acknowledging the issue and taking the time to respond.  Have you looked into other options in terms of web hosting?  You don't need to *buy* hardware or own your own servers in the modern information age, there are lots of hosted options that are faster and cheaper.  Microsoft's Azure and Amazon's Web Services can likely outperform any hardware Avsim has access to. 

[Chase 303]

We did consider these avenues. We analyzed traffic and bandwidth and determined that we use on the average of 10 terabytes of data each month, with peaks that go even higher. With our current set up at a secured facility near DC (can't get into too many details but it's literally a massive server site, or as I call it a guarded fortress, that Tom got a very competitive deal on many years ago) and our existing rack of servers and a SAN, we chose price stability over cloud based hosting. Furthermore, it would be so wasteful to not use the expensive set of servers we have already if it still works. Sure we had to swap out some aging HD's and opted to buy a SAN, but versus buying new servers or cloud hosting, it seemed much more predictable and stable. I firmly believe once we get configured properly we will be just fine.

[n4gix 4,947]

You don't need to *buy* hardware or own your own servers in the modern information age, there are lots of hosted options that are faster and cheaper.  Microsoft's Azure and Amazon's Web Services can likely outperform any hardware Avsim has access to. 

 

Chase already addressed this, but a quick search for "Azure" will find the reply from Tom himself about the costs for Azure:

http://www.avsim.com/topic/436818-so-what-happened-to-avsim/page-3

 

 

Go to the MS Azure site and price out a comparable Linux based system. When I looked, their Linux based solution was over $4,000 a month before you priced in bandwidth and database size. Factor in 11 Terra-bytes a month and the size of the database and I shudder to think what the monthly cost would be. I can say this with certainty... The community would not be willing to pay for it, that's for sure.

[jbinner 15]

I am far removed from the current discussion above then I used to be, but rest assured Chase and the team working the IT side of things have taken what we (I) installed in 2009 and made it VASTLY better.

 

NO online service is going to be able to touch the stability and sheer volume of data traffic as the current  NOC does. It is quite literally secure from a physical and data point of view. Not Azure, not rackspace, not godaddy, not google, not anyone. 

 

Comparing the sheer volume of AVSIM's library and forums to ANY other site it like comparing an apple to a zucchini. They have the right people and server farm in place. Please just give them some time to do their job AND their volunteer work here.

 

--Former IT Staff, AVSIM