System Woes and Warts - A little background

Over the last six months there have been a number of days that resulted in our community members pulling their hair, gnashing their teeth, and calling us (and me specifically) all kinds of names because of system outages, time outs, ad nauseam. I will be the first to admit that we have been struggling from time to time with the system. But, I am getting ahead of my self. Let me review a little history.

When we first setup AVSIM in our collocation service (known as a COLLO service), the architecture consisted of a single MySQL database. That serviced primarily the file library and was installed on JUPITER, our library and email server. When we brought online our new forum system in 2006 (the first generation of the one you see today), we decided to host it on a second server, MARS. But because we needed a MySQL database to run it on, and our previous forums were simply text based systems, we decided to simply connect the forum to our existing MySQL database on JUPITER. That worked for a long time; until the last six months to be exact. Seven years of successful service via an architecture that we did not anticipate would have a load placed on it that we see today.

Let's set that aside for a moment and address another issue. As I write this, over the last 24 hours, we have had over 670 spam attempts from one country alone, and not even one of the "biggies" in terms of spam sources. We have a number of functions in place that block spammers, including the service known as "Stop Forum Spam" or SFS for short. But, that doesn't prevent attempts to register - in order to block spam, we need to at least get a bit of information before they are blocked (email, IP, etc.). That takes server time to accomplish - server time that is taken away from you as a community member. Add all the spam attempts over a 24 hour period and that is a tremendous amount of server time being used to protect you and this community.

Okay, so, one more... if you look at the bottom of the main forum index page, you will see a breakdown of members and "guests" online. "Guests" take three or more forms. The first is that they are indeed users who have not registered or have not logged in and are here viewing the forums and its content. The second form of "guests" is that of search engine search attempts. Every time someone uses Bing or Google or any of the couple of other dozen search engines are used to find something that can be found on AVSIM, the system registers the search effort as a "guest" visiting AVSIM, which indeed it is.

The third form is that of "spiders" or "bots" that sweep through our servers looking for data to put in search databases, or for other, not so positive acts; like collecting email addresses and any personal information they can find. That process consumes huge amounts of server time and Apache connections - again stealing those from our legitimate members. There are hundreds of these and many of them are not "friendly" like Google and Bing are. The fact of the matter is that we have processes in place to prevent the bad bots from dragging the system to its knees. However, as in warfare, there are bots that masquerade themselves, use well thought out "spoofing" methods and successfully avoid or circumvent our protective measures.

Finally, there are features that we employ that do add significant load to our servers. Take TapaTalk as an example. It multiplies server load by 5 times! We have been working with TapaTalk on this, but if no solution is found, we might have to remove it entirely, and never look back again.

Now let's go back to the architecture... Because we rely upon one database between two servers, we have a bottle neck that exists between the two. That bottle neck is the interface required to go from MARS to JUPITER in order to connect the forum with the MySQL database server. In 2011, 12 and most of 2013, that was adequate. In fact, we rarely, if at all, suffered problems because of the single MySQL implementation. That has obviously changed. I guess you could say that we are suffering from our own success. We have out grown our configuration and we need to change it to prevent further "throttling" of performance for our registered community members.

So, how are we going to fix this? We are working to bring on MySQL experts and Linux / Server / Apache gurus to focus on the issues and we are daily taking remedial steps (like blocking all but a handful of guests at any one time) to make the AVSIM experience as positive as it can be for you.

It is our hope that your patience will withstand the time outs and outages, and that the outcome will be more than sufficient to make all these to be nightmares of the past.