The AVSIM HACK of 2009
Originally Published in Tom Allensworth's Blog
June 10, 2012
Last Updated: April 16, 2013
Talking about things that AVSIM Staff are proud of, how many of you remember Denver, Colorado, United and the UA training facility there? How many were in attendance? It was a first of its kind for the simulation community. United, despite the heightened concerns about security resulting from 9/11, sold us over two hundred hours of full motion simulator time at a price that was, well, charitable. We only found out after the fact that the time they sold us was actually 20 percent of what they normally charged airlines other than United for the same access.
With that positive memory firmly in hand, we come to a very horrible, but uplifting time for AVSIM and a truly shining moment for the flight simulation community - the 2009 AVSIM Hack. I have hesitated to give more details until now, simply because AVSIM picked up and moved on, and I did not feel there was a need to do so. I have been scouraged by some in the community for not having done so (as in, we had something to hide) and I am reminded by my colleagues that in the interest of history, and long after we are gone, people will talk about this event and possibly see it as a water shed occurrence in our hobby. I don't know about that - you are welcome to draw your own conclusions.
In 2008, Matt Johnson, our tech manager and all around IT guy for AVSIM, made it known that his real world responsibilities would keep him from putting in the time that he had unstintingly provided for many years. He also told us that he could not guarantee that he would be around to help us out if things went "south". He was leaving the University environment in which he had worked from graduation, and going to work in the commercial world. The game play had significantly changed for him. Obviously, the responsible thing for us to do was to find a replacement for Matt, and do so rather quickly. I can tell you that replacing such a talent is not an easy thing to do, especially as a volunteer, which Matt was.
Our servers were getting long in the tooth, and our risk assessment of them was growing more ominous by the day. In fact, in the fall of 2008 a disk sector had gone bad. We took that as a ominous harbinger of things to come. What we did not know was the future role that bad disk sector would play... THE single most significant role in saving AVSIM.
In the latter part of the year, we brought aboard Stan Harmon as a paid consultant for his professional experience in Linux and its implementation. We asked John Binner, who was our hardware manager, to work with Stan to define a remediation plan for our existing hardware and operating systems. In early 2009, Phil Dawson appeared on the radar as a potential designer for what we hoped would become our new library. In January of 2009 he had opened his own IT business, SOURCEWISE LTD. in the UK, and was very much involved in the flight simulation community via his website; SIMFLY.EU (see image below). It was the SIMFLY website that established his credentials in our thinking and we invited him to join our team.
A lot was happening all at once. Microsoft had just announced the closure of the ACES Studio. The global community was in an uproar as to the future of flight simulation, and in the storm that surrounded of all of this, we were looking to both address our immediate hardware issues as well as set the stage for the redesign of AVSIM. Our first priority, in terms of design was the file library and we asked Phil to look at that. In the meantime, Stan and John were working out the hardware issues.
The shuttering of ACES precipitated a call for a meeting of industry to discuss the future of flight simulation and the direction that we should take given the demise of the MS Flight Simulation genre'. We decided upon a meeting to be held at a hotel adjacent to Schiphol airport in the Netherlands in April. Since I was going to be there, I decided to fly Dawson over from the UK to meet me to discuss the file library and its conceptual design. Phil agreed and met me at the hotel. Leading up to the meeting at Schiphol, we had given Phil moderator access to our forums, where he had also volunteered to help.
Phil and I discussed the library, what our larger vision of the library was, and what we had hoped to accomplish in its redesign. Phil returned to the UK and we went ahead with our meetings regarding the future of flight simulation.
The timeline from this point until the evening of the 12th of May gets a bit confusing, but here is the long and the short of it. Phil asked for increasing access to our servers to fulfill his role in the redesign of the library. He also volunteered to help us with the hardware issues we were having. We gave him the access that he argued successfully that he should have. That was a tremendous mistake of judgement that we would come to regret. In our defense, there were no indications that Phil could not be trusted. Who would expect a volunteer to have anything but honorable intentions? Well, we found out the hard way that not all volunteers are what they appear. That has had lasting implications.
In the day or two leading up to the hack, spam email had been sent from SIMFLY.EU to every member of the AVSIM forums. AVSIM members started reporting that the spam was hitting email addresses that they only used to access AVSIM. It didn't take much to conclude that Phil had taken advantage of his position and had stolen AVSIM's forum email database. Phil eventually admitted as such in the forums, after the hack.
On the evening (EST) of May 12th, we removed Phil's access to the admin functions of the forums and we started to shut down his access to the workings of our servers. Stan and I were both online, communicating, and watching the servers when we noticed that our directories were starting to disappear. We were not fast enough. At about 3 a.m. UK time (10 p.m. EST), Phil was attacking our servers and doing a data refill of our disks.
In an apparent fit of anger resulting from our removal of him from our forum administration, Phil went on a rampage, attempting to wipe out the entire AVSIM structure. He knew about our bad sector and our weakening disks. He apparently believed that he could get away with wiping out our system and blame our tottering architecture to cover his tracks. He had also set up a trip to Turkey which he was leaving for early on the 13th. He attempted to later use both as an alibi. The Turkish trip is substantiated in later court testimony. See the link below.
Phil was a smart guy, but not that smart. He had set an automated process to run which would delete disk sectors, fill them with "zeros" and effectively render them unrecoverable. What he did not anticipate was that the very disk sector that was bad and which we were all painfully aware of, would be his undoing. That bad disk sector caused his automated process to come to a screeching halt. When that happened, his process stopped short of destroying our access to the logs as well. As result, we had a log of everything that Phil had done, starting in the evening of the 12th Eastern Standard Time. His fingerprint, portrait, and genetics were all over the attempt to wipe us out. The logs told all.
Stan and I watched this unfold that night from our PC's in absolute shock. We could not shut down the process fast enough. Luckily, it hit the bad sector, died, and not known to us at that moment, we were saved. But we did not know that until much later. At approximately 2300 on the 12th of May, I sent out an email to the world stating that we had been hacked and that it appeared we were destroyed. Nothing we could see at that time indicated any possibility of reviving AVSIM. That was the worst moment in the history of AVSIM - and one that I was sure we would never recover from. Every indication was that we were dead. Phil had known enough about our system to also know of our backup server and he went after it too. From what we could see that night, AVSIM no longer existed.
It was not until after Stan made multiple trips to our Network Operating Center (NOC) in Northern Virginia, that we were able to really establish the amount of damage and our potential for recovery. Phil's ignorance of the impact of the dead sector set the stage for us to recover far more than we had ever hoped possible.
The following morning we set up a temporary forum elsewhere and started to provide forum services once again to our community. Almost immediately, the user community launched a donation effort. We were frankly surprised by that, and in short order we recognized that the community stood solidly behind AVSIM. As our understanding of the depth of the community's dedication to the AVSIM cause deepened, we opened a PayPal account to accept those donations. I think I can safely say that the AVSIM Team was astonished by both the community initiated donation effort and the results of that effort. The community raised well over $30,000 USD to help get all of AVSIM back online. To say that we were shocked and pleasently surprised by this still remains an understatement.
Our initial plan had been to effectively give into the hack, pick ourselves up, and wipe what remained of our existing disks clean and start over with the aged hardware then in place. As the community's contributions continued to grow, we reassessed that plan and decided that given we were able to recover data, and that we wanted to continue, grow, and extend our services to the community, it made sense to do our very best to put in place hardware and systems for the future. That's the plan we finally adopted and which you see the benefit of today.
As the tech team moved forward with getting our hardware and systems back online, we decided to pursue Phil for violating a number of UK and US laws and to recoup damages from him, both monetarily and otherwise. Those were inconsequential compared to our larger concern. The most important motivator for us was that Phil was attempting to sell himself as an IT guy who would set up and manage commerce systems for business owners in the UK and elsewhere. Given what he had done to us, the thought of Phil controlling the back end of commerce systems with access to financial and credit information was the last thing we wanted to see him get away with. We could not allow him to wreck havoc with an unsuspecting business owner and the customers that faced the potential of fraud. As a result, we hired a prominent international law firm, London based law firm K&L Gates LLP, and went after Dawson. K&L Gates also had U.S. offices that we could use if we decided to pursue U.S. prosecution.
I met with the London police in November of 2009 to provide data and a deposition which they would then hand over to the authorities in the appropriate jurisdiction. K&L Gates had numerous contacts with various departments and jurisdictions, including that of Phil's home town police. It became apparent that law enforcement in the UK did not consider this incident sufficient to prosecute, nor that the risk of fraud warranted pursuit. Given that realization, our choices were to spend enormous amounts of money to pursue a civil case against Dawson or file a U.S. complaint. Neither of these choices were guaranteed to succeed and would have cost well over 100,000 UKP ($180,000 or more at the time).
At the point that we shut K&L Gates down, we had spent over $10,000 of AVSIM's savings. We had not spent any of the community's contributions on the effort. Everything that the community had donated toward our recovery effort to that point had been spent on hardware, software and the consultation fees to get us up and running with the new hardware.
We decided that investing in the community was a better decision going forward and certainly a better use of AVSIM's funds. That decision was effectively made at the end of 2009 after my meeting with the London police and further consultations with K&L Gates. Our decision was that it would be far more beneficial to the community to put our money and that of the community's contributions to work directly, rather than pursuing a legal case that would do nothing to advance our hobby and accomplish nothing more than incurring futher expense.
Ultimately, the outcome was that we put into place multiple high performance servers with raid arrays, extrodinary amounts of RAM, and plenty of room for expansion. On Memorial Day a couple of weeks ago, John installed a high performance background backup system to further insure that anything short of an internal hack could rapidly be overcome.
This story was a tragedy on a number of levels. If you visited the Simfly.eu site link above, you will have seen a hint of what transpired with Phil. Without further comment, we will direct you to this link to conclude his part of the story: http://www.thenorthe...ver_was_busted/
I would like to pass on a couple of thoughts...
Phil was an impressive young guy. My meeting with him left me confident that he could do the job we were asking him to do, and I had no reason to question his integrity or sincerity. I was, and continue to be, stunned that he chose to do what he did. I cannot begin to tell you how frequently I have questioned my judgement in people then and since.
The greatest moment in AVSIM's history is and will remain the willingness, initiative and commitment by the flight simulation community to stand up and provide us the means and where-with-all to put AVSIM back together. It was a very humbling moment for all of us on the AVSIM team. We cannot begin to convey our thanks to all that contributed to our resurrection and those that have contributed since. Our gratitude is without bounds, and we cannot say thank you enough. Thank You!