Sign in to follow this  
TechguyMaxC

Avsim down time

Recommended Posts

I would like to give the site administrator(s) my feedback about the frequency and length of down time for this site.  I intend the following to be inquisitive/informative in nature, and not a criticism of any person or organization.  

 

I visit approximately 20 web sites per day, several of which are forums.  None of those sites experience the number of down time incidents as Avsim.  Also, when the site does go down, it seems to remain down for up to several hours.  This seems to have gotten worse in the past month or so, but as a visitor to Avsim since the late 90's (when it played host to the MSFS Benchmark sub-site and forum), I can say that the number of "outages" over that time is far more than any other site I have visited so it is not merely a recent phenomenon.  

 

I am curious as to the specific cause of this issue.  Is there too much traffic for the server(s)?  Are the servers owned by Avsim or provided by a web host?  Are they co-located or do you own the building in which they are located?  What are the specs of the servers and how many are there?  How much bandwidth do you have access to?  Peak simultaneous user count?  Average users per 24/hours?  I believe with answers to these questions (and a few others) the problem(s) can be solved.  If you need advice relating to technology, I am an IT professional and would be more than happy to offer my services to consult on the matter at no charge.  If you would like to speak with the owners/administrators of other communities that I am involved in, I can put you in touch with them as well.  

Share this post


Link to post
Share on other sites
Help AVSIM continue to serve you!
Please donate today!

Avsim forums have been more reliable (for me) since they made upgrades to their hard disk drives a few weeks ago... until today. Today it's been intermittent service since about 1030 GMT up until about 1530 GMT, with a variety of "Unable" and 502 errors.

 

Previous to those HDD upgrades Avsim went down every day with 503 errors from approx 0830 GMT to 1000 GMT (this has been covered elsewhere in great depth so I'm not going to detail that any further, suffice to say it seems fixed now)

 

Ultimately though, I have to agree with the OP; Avsim has by far the worst uptime of any website I've regularly visited over the last few years. I do appreciate this is a free-to-use service run by volunteers, but the facts regrading uptime remain.

Share this post


Link to post
Share on other sites

If its a money issue then perhaps have monthly targets with a live counter going, it seems to work for a lot of websites. Tapa talk would be kind of nice to have as well

Share this post


Link to post
Share on other sites

I've personally never seen a 'donation' site with a board of directors, either. With that sort of formality, there should be a much more reliable environment than is present today.

Share this post


Link to post
Share on other sites

Personally, if every single member gave $1 a month as a subscription fee, that would more than pay for enough people to maintain this site and less issue with downtime.  I would be more than happy to pay that.

 

-Jim

Share this post


Link to post
Share on other sites

Personally, if every single member gave $1 a month as a subscription fee, that would more than pay for enough people to maintain this site and less issue with downtime.  I would be more than happy to pay that.

 

-Jim

 

I'm sure lots of people would be happy to contribute to the site.  Let's determine the problem before we throw money at it though.   :wink:

Share this post


Link to post
Share on other sites

I'm sure lots of people would be happy to contribute to the site.  Let's determine the problem before we throw money at it though.   :wink:

Isn't that what the board of directors is for? :D

 

-Jim

Share this post


Link to post
Share on other sites

Thank you everyone for expressing your concerns.  Please keep in mind the following (and yeah by now you're saying we've heard this before, I know):

 

  • We are a donation based website with some supplemental income from FS advertisements
  • We are staffed entirely by volunteers
  • Our IT services are provided pro bono by an IT guy who has a demanding full time job
  • Each of us has a full time job outside of AVSIM and devote a lot of our free time to the site away from our families

We are on a path to bringing the site where it needs to be.  Why do we have seemingly constant issues lately?  The honest and most humble truth is, we don't have thousands of dollars laying around to pay someone full time to do the work.  Also, we are working with aging equipment that is still useable for the foreseeable future but requires some "refreshing" and tweaking.  Basically, we are doing load balancing between two servers, and it's something AVSIM has never done before.  Sure, we used to have our library and forum on two separate servers, but we have true load balancing happening now where both servers are sharing loads and talking to each other to serve one product (also add a very new and shiny SAN device in there between the two).  Because of this change, there are some hiccups here and there.  Small ones that get fixed as soon as we notice/hear about them, and as soon as we can get our volunteer IT guy in to fix it.  We have a professional grade IT equipment, with part time IT help.  It's help we are most grateful for, but is limited.  You all remember what happened the last time someone offered us IT help when we needed it most -- we were nearly destroyed.  I don't call it paranoia, but we do call it extreme caution when we have hand selected an IT guy that's been with us for a long time.

 

We do bits and pieces of work as we can, little by little.  Our IT guy has spent several of his entire weekends (including a holiday weekend) at the server location doing diagnostics, installing new hardware, etc.  On his time off he's working on the software side of things trying to get things where they need to be.  This has been a rather expensive endeavor (already exceeding the entire year's donations), and we appreciate your support and understanding.

 

Some of these whacky conspiracy threads around the internet might lead you to believe we are just raking in the dough, it's the farthest thing from the truth.  We barely make ends meet each month to pay the bandwidth expenses alone.  Tom built AVSIM has a place where we would be the voice of the community without being a commercial "for profit" website.  I don't see us ever changing that, because that's not what Tom's idea was about.  

 

Regarding the what I consider slightly cheeky but also kind of "punch in the gut" comments about the board of directors... Please remember this title does not mean we make money... No one makes money at AVSIM and never has (please see paragraph above).  There were some months when Tom would pay for things out of his own pocket.  I know for a fact Tom ended up footing a good chunk of the expenses of our last convention just because the show must go on and he always put the community first--that was the just the kind of guy he was.  He would never want anybody to know it though.  The board of directors was a group of people Tom hand picked to keep his legacy alive and has the responsibilities of overseeing the operations and decisions since his passing.  

 

I speak as a volunteer of AVSIM today, a volunteer that loves this community and loves this website.  I have been a volunteer for AVSIM since I was 14.  In many ways, AVSIM is a part of who I am because I've known it since almost as long as I can remember.  So please, know that the times are not easy right now, but we are working forward to a better tomorrow.  Bit by bit, piece by piece, things are going to get better, as they already have... Performance wise, I think we are lightyears ahead of where we were only months ago (notice page loads aren't 12 seconds anymore).

 

Thank you, sincerely, again for all of your support... I do see very good things for AVSIM in the future with the trajectory we are on and the wonderful community we have supporting the efforts along the way.

 

Chase

Share this post


Link to post
Share on other sites

Hi Chasse,

 

A very professional response to the community members which should discourage further criticisism of volunteer work by a few members which benefit all of us. Regarding required operating funds, I would support a yearly fee proportional to the average yearly shortfall. Food for thought.

 

JJ

Share this post


Link to post
Share on other sites

Hi Chasse,

 

A very professional response to the community members which should discourage further criticisism of volunteer work by a few members which benefit all of us. Regarding required operating funds, I would support a yearly fee proportional to the average yearly shortfall. Food for thought.

 

JJ

These offerings of support always go unanswered and Im sure they have their reasons for not going down this path.

Here is is good example of it in action, go to the top right for the counter  http://vattastic.com/

Share this post


Link to post
Share on other sites

Chase, thanks for acknowledging the issue and taking the time to respond.  Have you looked into other options in terms of web hosting?  You don't need to *buy* hardware or own your own servers in the modern information age, there are lots of hosted options that are faster and cheaper.  Microsoft's Azure and Amazon's Web Services can likely outperform any hardware Avsim has access to. 

Share this post


Link to post
Share on other sites

We did consider these avenues. We analyzed traffic and bandwidth and determined that we use on the average of 10 terabytes of data each month, with peaks that go even higher. With our current set up at a secured facility near DC (can't get into too many details but it's literally a massive server site, or as I call it a guarded fortress, that Tom got a very competitive deal on many years ago) and our existing rack of servers and a SAN, we chose price stability over cloud based hosting. Furthermore, it would be so wasteful to not use the expensive set of servers we have already if it still works. Sure we had to swap out some aging HD's and opted to buy a SAN, but versus buying new servers or cloud hosting, it seemed much more predictable and stable. I firmly believe once we get configured properly we will be just fine.

Share this post


Link to post
Share on other sites

You don't need to *buy* hardware or own your own servers in the modern information age, there are lots of hosted options that are faster and cheaper.  Microsoft's Azure and Amazon's Web Services can likely outperform any hardware Avsim has access to. 

 

Chase already addressed this, but a quick search for "Azure" will find the reply from Tom himself about the costs for Azure:

http://www.avsim.com/topic/436818-so-what-happened-to-avsim/page-3

 

 

Go to the MS Azure site and price out a comparable Linux based system. When I looked, their Linux based solution was over $4,000 a month before you priced in bandwidth and database size. Factor in 11 Terra-bytes a month and the size of the database and I shudder to think what the monthly cost would be. I can say this with certainty... The community would not be willing to pay for it, that's for sure.

Share this post


Link to post
Share on other sites

I am far removed from the current discussion above then I used to be, but rest assured Chase and the team working the IT side of things have taken what we (I) installed in 2009 and made it VASTLY better.

 

NO online service is going to be able to touch the stability and sheer volume of data traffic as the current  NOC does. It is quite literally secure from a physical and data point of view. Not Azure, not rackspace, not godaddy, not google, not anyone. 

 

Comparing the sheer volume of AVSIM's library and forums to ANY other site it like comparing an apple to a zucchini. They have the right people and server farm in place. Please just give them some time to do their job AND their volunteer work here.

 

--Former IT Staff, AVSIM

Share this post


Link to post
Share on other sites

I remember many years ago that Tom posted several pictures of "The Rack" at the NOC. It's a shame I cannot seem to find them now...

 

...as it was truly impressive seeing that robust and expensive server rack!

Share this post


Link to post
Share on other sites

AVSIM does an amazing job of maintaining this website. Complaining about it just doesn't make sense to me. No other website I can think of has the sheer volume of traffic that this one does.

 

I'd say the AVSIM team is doing better than most and deserve kudos not complaint. That's my opinion.

Share this post


Link to post
Share on other sites

I think we all probably get a bit frustrated when AVSIM is down, simply because it is a very important part of our lives! Nevertheless, we need to retain perspective where maintenance and running costs are concerned. Like Ed has already stated, the volume of traffic through AVSIM is huge in comparison to any other flightsim website, and is run by enthusiastic volunteers who spend a lot of their free time and effort keeping it operational for the benefit of the entire flightsim community. I am extremely grateful to all of you, and I will be making another donation to the site as soon as I get home from work.

Share this post


Link to post
Share on other sites

I found a pic from a recent visit to add the new disk array, the beating heart of Avsim . 

Thanks for the more recent photo, Chuck. Perhaps the photo should be placed as a sticky somewhere to show to the next person who is inclined to think AVSIM is run on a creaky and ancient 486DX computer from somebody's bedroom... :LMAO:

Share this post


Link to post
Share on other sites

Great and well said #9 post Chase.   I brought the Subscription thing up last year before Tom passed,  i thought maybe it was a great idea at the time but Tom did explain it different and then it made more sense.  People you can donate do and folks who cant, cant,  everybody gets to share the beautiful hobby.  

 

PS,  the site is flying along now  :smile:

 

 

 


I'd say the AVSIM team is doing better than most and deserve kudos not complaint

Correct 

Share this post


Link to post
Share on other sites

AVSIM does an amazing job of maintaining this website. Complaining about it just doesn't make sense to me. No other website I can think of has the sheer volume of traffic that this one does.

 

I'd say the AVSIM team is doing better than most and deserve kudos not complaint. That's my opinion.

 

Complaining and seeking information to offer potentially helpful suggestions based on expertise are two very different things.  I made no criticism of anyone, and even prefaced my remarks to indicate that.  There are lots of sites that have much larger forums than Avsim.  Anandtech, for one.  Check their user count here: http://forums.anandtech.com/index.php (bottom of page).  I've been a member there almost as long as I have here and the number of times that site has gone down unexpectedly can be counted on one hand.  Even this relatively small community also run by volunteers has less down time than Avsim, and their user count isn't too far off from Avsim's: http://www.rage3d.com/board/index.php  It is undeniable that Avsim has high volume, it is also undeniable that they have a high frequency of issues.  Are the two related?  Almost certainly.  Are other large sites able to operate without frequent down time?  You betcha.  So what does that mean?  There is a solution.  The question is, is Avsim willing to do what it takes to solve it?  Time will tell.      

Share this post


Link to post
Share on other sites

AVSIM does an amazing job of maintaining this website. Complaining about it just doesn't make sense to me. No other website I can think of has the sheer volume of traffic that this one does.

 

I'd say the AVSIM team is doing better than most and deserve kudos not complaint. That's my opinion.

 

I can think of one - I used to be responsible for its design and performance. We did around 26 terabytes a day, more on busy days.

 

AVSIM gets a pass on the tech front because they are the one-eyed man in the land of the blind. If they're still buying hardware and (gasp) a SAN in 2015, I'm at a loss for words. There are plenty of multi-million- and billion-dollar businesses that trust their infrastructure to the cloud and wouldn't consider owned infrastructure. I can think of three major ones off the top of my head.

 

Techguy's last posts make sense - the forums aren't particularly large relative to other forums and the file library, while it accounts for much of the bandwidth, is relatively simple and light to process. There are thousands of sites that have more significant loads than AVSIM who can handle it quite well without issues.

 

Cheers!

 

Luke

Share this post


Link to post
Share on other sites

 

 


The question is, is Avsim willing to do what it takes to solve it?  Time will tell.

 

That has never been the question.  I think it's quite clear we are doing everything we can.

Share this post


Link to post
Share on other sites

There are plenty of multi-million- and billion-dollar businesses that trust their infrastructure to the cloud and wouldn't consider owned infrastructure. I can think of three major ones off the top of my head.

Luke, there is a vast difference between "multi-million- and billion-dollar businesses" and a not-for-profit "business" running on a shoestring budget and staffed entirely by volunteers.

 

That should be self-evident. Your comparison is a total non-sequitur.

Share this post


Link to post
Share on other sites
Guest
This topic is now closed to further replies.
Sign in to follow this