More on the Hack
By Tom Allensworth
This was originally posted on the temporary forums that we set up after the hack.
Originally published on May 28, 2009
Just over two weeks ago Black Tuesday struck and AVSIM as we knew it disappeared. As Stan, one of our tech consultants, and I watched via our secure connections to the AVSIM system that evening, we saw it crumbling before us. Connected via the phone and with silence ringing in our ears, Stan and I were speechless in disbelief. Over twelve years worth of work, tens of thousands of man hours, the contributions of thousands of AVSIM staff and FS enthusiasts, a community in every sense of the word... all gone. Or, so we thought.
There was no doubt then, and no doubt today, that AVSIM was taken out by a "hacker". I put hacker in quotes because I am not sure that is the correct term to use. Who would deliberately and with malice aforethought do this? More importantly, why would someone do this? But... more on that subject later.
As that evening unfolded, we thought we knew what had taken place. Since then, with the help of some world class technical support from literally around the world, we have been able to delve into the details and are now able to state unequivocally the sequence of events and ultimately, the deeds that brought AVSIM to its knees.
The hacker appeared to have attacked the PURPLE server; that is, our web and forum server first. We saw functionality start to disappear; CGI driven functions initially, and then the entire server fell silent. It was gone. We subsequently determined that he had executed a complete over-write of the PURPLE system, effectively destroying our web and forum server.
Stan and I then hopped over to GREEN, our library and email server and again saw the tracks of intrusion. Stan immediately instructed GREEN to reboot, but the damage was done. The partition table had been erased. No partition table = no boot disk = no online system.
Our immediate conclusion was that all of AVSIM had been destroyed, and that was the message that we immediately issued to our community via Nels Anderson at Flightsim.com and Francois Dumas at Simflight.com - historical competitors of AVSIM's, but staunch allies and supporters of the Flight Sim community, including the AVSIM community. The word was also sent out to all of the International Flight Simulation Consortium members, who in turn posted the word on their various web sites and forums. Thanks to the efforts of all of these fine people, by noon on Wednesday, the 13th, the world knew that AVSIM was offline, maybe for good.
We would not know for sure until we could get Stan to the Network Operations Center (NOC), where our servers were located - and that was not going to happen until the weekend of the 16th - four days after going offline.
On the morning of the 13th we were able to bring online a temporary forums system using a site that I had been experimenting with for another interest of mine. Again we put the word out via Nels, Francois and IFSC, and people started to flock into the new forum system.
What rapidly became apparent was that the flight simulation community wanted to help. Help in anyway they could; donations, technical support, forensics support, security engineering support; you name it. It was an impressive turnout by the community - people volunteering to bring their talents and knowledge into play, in order to get us back online. We were bombarded by email and forum offerings and I have to say, that outpouring of support both surprised us and helped convince me that not to bring back AVSIM was not an option and the Board of Directors emphatically agreed.
The question remained however - bring AVSIM back online in what form? With what content - did any content remain? Or would we have to start all over again? We would not know until Stan could lay his hands on the system, but we knew one thing for sure... We were coming back even if we had to start over from scratch.
GREEN and PURPLE were now "crime scenes" as far as we were concerned. That meant that to come back, we had "bad, very bad, or worse" choices available to us. No matter what choice we made, given as we believed that we had lost all of our backup data, we were going to be offline for a while to come:
1. We could keep the site down until both systems were forensically analyzed and the data recovered from across all installed disks, if at all possible,
2. We could purchase new servers, giving us the time to do #1, and start all over with the hope that we could recover some legacy data to repopulate the site with,
3. We could reformat and reload the OS on both servers, bring them online in a "start over" mode, and in doing so, destroy any evidence of a hack,
We held our breath through the weekend of May 16 and 17, awaiting Stan's inspection and report out to us. On Monday morning, the 18th, Stan briefed us on the status of PURPLE and GREEN and we looked again at our options. Stan's visit allowed us to run a special tool which would hopefully allow us to re-establish the partitions on both servers. That was not to be.
PURPLE was completely gone. There was no way to recover from the partition table deletion and we did not know at that point that the hacker had over-written the entire disk, making any chance of on-site recovery impossible.
GREEN on the other hand showed some partitions that were recoverable! The first good news we had received out of this disaster. What that meant was that there was the possibility of recovering significant amounts of data as GREEN connected into our Network Accessible Server (NAS), where both servers were backed up and where the library files resided.
The realization that we might have significant amounts of data saved from the hack, the overwhelming financial support of the community, and our overwhelming need to bring AVSIM back as quickly as possible made our decision easy. On Monday afternoon the AVSIM Board of Directors unanimously voted in favor of using the community's contributions along with some of our savings to purchase two new servers. This would allow us to get back online quickly, provide for future growth, and provide the scalability, security and topology to never have to go through this again.
On Tuesday morning, the 19th, we ordered the servers along with a Gigabit Switch for delivery at the NOC on Thursday afternoon. The servers were delivered on time and Friday slowly approached.
(THIS WAS POSTED IN ANOTHER ENTRY ON THE SAME DAY)
THE CREW THAT BROUGHT AVSIM BACK
It is not often that we bring particular attention to AVSIM Volunteers (they normally get big BY LINES and a pay increase), but the events of the last 16 days demand that we do so. The AVSIM Editorial Staff would like to put the shine of the public spotlight on these hard working guys, so that you know the team that put AVSIM back together again.
Matt Johnson: Matt joined AVSIM in 2000 and brought a technical genius to our team that we never dreamed we would ever have. Matt designed the AVSIM File Library system from scratch as there were no "off the shelf" library software available at the time that would either meet our needs or survive under the load that the community would put on it. Matt was AVSIM's Technical Manager up until the middle of 2008 when his real world job took all his attention and time away from AVSIM. On Wednesday, May 13th, the morning after the hack, Matt stepped up and volunteered to help us bring the AVSIM system back online. Living near London, England, Matt worked his magic in support of the other team members from afar and was critical to the system being back online today.
John Binner: John is a former U.S. Marine Corps Air Traffic Controller and AVSIM's hardware guy. John had been very involved with the set up of the IPB forums back in November, 2008 and throughout the Christmas break. Before that, John spent countless hours making sure that the staff did not do a "FUBAR" on the system. John was the "go to guy" in all things related to the system and other critical management functions at AVSIM. John has also been a key contributor to the success of the AVSIM FANCONS, starting with our event in San Diego way back when. John left AVSIM earlier this year, but when the hack occurred, he stood up and volunteered to help us get the system back online. John volunteered to go and do whatever it took to get the AVSIM system back and in operation, including sacrificing the entire Memorial Day weekend. He spent the entire holiday camped out in Herndon, Virgina - a long way away from his family and home in South Carolina.
Stanly Thompson-Harmon: What can we say? Stan is the consummate geek. Were pocket protectors still in vogue, Stan would probably own one for ever day of the week (we think he has a couple of dozen stashed in his desk but is afraid to show them). Stan is an engineer for a top two defense firm in the U.S., and breaths and speaks computer code. Stan has put in countless hours since before the hack, looking into the issues we were having with GREEN's bad partition. Stan sacrificed his memorial day weekend too to get AVSIM back in a condition that would allow us to get the system online this week.
The Un-Named One: For security reasons, we are unable to name a special assist from a person or persons in recovering the data on GREEN, PURPLE and the NAS. With their special forensics talents, AVSIM was able to restore all of the web site, all of the forums and we anticipate 90% of the library including all FSX and FS9 files. We expect to be able to serve files dating back to FS98. Thank you UNKNOWN GUY for your assistance and skills.
Matt, John and Stan came together and put humpty dumpty back together again. With the overwhelming help of the community, AVSIM is bigger, better, and faster than ever, and now armed with a system ready to tackle the future growth and expansion of the AVSIM community. From all of the AVSIM staff and a couple of thousand AVSIM Community members, thank you guys! Bravo Zulu!