Is AVSIM undergoing a Denial Of Service attack?!?!?

[Luke 782]

 

 

index.html, I have no idea where that's going because we use .php...

 

You probably want to dig into it. It takes 15-27 seconds to render the page on the server each time it is requested. What is your database doing during that time? What effect does it have on other requests?

 

15-27 seconds on a simple GET request - that's a recipe for a successful DOS as it takes orders of magnitude more resources to respond than to request. A half-dozen bash scripts running wget could bring things down?

 

Cheers!
 

Luke

[codechris 9]

Good find there Luke.

 

Yes, AVSIM, you really want to clamp that down. If you don't want people doing to index.html then block or remove that page. There are several ways of doing that and all pretty easy. It could be that if several people are hitting that page, you effectively have a lot of users putting unnecessary strain on the servers.

[DarkstarF16 72]

Probably over a year ago I contacted Tom regarding this issue. I live in the UK and at the time was getting the HTTP 503 error between 8am and 9am UTC daily. Toms reply was... "I have tried to tell folks in both the UK and central Europe; you have DNS issues along the way to AVSIM. There are at least one DNS system both in the UK and the Netherlands that are causing no end of issues with things like you are seeing. I have no control over those."

 

It's APPARANT that the DNS issue is still ongoing because now between 9am and 10am daily like clockwork I am unable to access AVSIM.

 

I have since moved to a completely different location and using a different ISP. However, most if not all the ISP's in the UK probably route though a common DNS server which is still causing issues. Just thought I'd share.

If I can do anything to help let me know.

[Chase 303]

I think once we get things squared away with our IT with a little help from our original designers, I want to research and look at content delivery services which would (I presume) mirror our server in different locations around the world so we don't have these issues.

 

The Board of Directors main objective is to keep AVSIM alive and thriving.  Growth is foreseen and we want to be able to meet the demands that entails.  DarkstarF16, once we get IT situated, let's keep in touch so perhaps we can figure this out together.

 

Tom lived to see AVSIM turn 18 on March 23.  Knowing how much he loved this community, I'm grateful he was here to see it before he left us for his journey home.

[DarkstarF16 72]

Sure, just PM me when the time is right.

[codechris 9]

Probably over a year ago I contacted Tom regarding this issue. I live in the UK and at the time was getting the HTTP 503 error between 8am and 9am UTC daily. Toms reply was... "I have tried to tell folks in both the UK and central Europe; you have DNS issues along the way to AVSIM. There are at least one DNS system both in the UK and the Netherlands that are causing no end of issues with things like you are seeing. I have no control over those."

 

It's APPARANT that the DNS issue is still ongoing because now between 9am and 10am daily like clockwork I am unable to access AVSIM.

 

I have since moved to a completely different location and using a different ISP. However, most if not all the ISP's in the UK probably route though a common DNS server which is still causing issues. Just thought I'd share.

If I can do anything to help let me know.

 

The 503 error is not a DNS error, also DNS is a little more complex then that and you can use any publically available DNS server. Try setting your computer to use a public DNS, google has two different IP's you can use, and repeat and see if you have the same results. You can also easily test if it is a DNS issue by, when the problem occurs, opening a terminal/command promp and type ping forums.avsim.net and see if it resolves the name to an IP address. If it does, then DNS is resolving fine.

 

The 503 HTTP error comes down to socket creation so for example the web server or load balancer is unable to open the socket so sends back a 503 error. 5XX errors donate server error of some sort. If this was a DNS issue you would expect a 4XX error, such as a 404. 500 donates connection with the web server has been achieved but not worked.

 

I've no idea what the AVsim setup is, but looks like traffic is routed through Vigina, so that puts 9am UK time at what...3am local? Could be an overnight job that's running cause issues. I'm purely guessing here. 503 more suggest high traffic load, but seams like an odd time. 10am European, ~3am US, ~6pm Australia time. Doesn't seam likely for high traffic.

 

And yes Chase Kreznor a CDN service does spread the load across nodes throughout the world, often sites use this for images as that can be where high load is but that's not the biggest issue for AVsim I presume. But, it would spread load, so it's not all concentrated in one area. Though load can be alleviated by adding more web servers to the pool etc etc

[Gartro 18]

That's interesting, especially the second-last paragraph, because I have historically only been able to access the Forum about 5% of the time if I tried at about 6pm Australian Eastern Standard time (UTC +10). I have learned to access before 5.30pm or after 7.30pm here.

 

Sometimes, I would get a 5xx error message but other times I would give up waiting and come back later.

 

Because I was well aware the time in eastern USA (one of my sons and his family had been living in Pittsburgh PA), I has assumed that the Forum was "down" at that early morning time each day for maintenance or similar.

 

Gary

[ckyliu 1,749]

Probably over a year ago I contacted Tom regarding this issue. I live in the UK and at the time was getting the HTTP 503 error between 8am and 9am UTC daily. Toms reply was... "I have tried to tell folks in both the UK and central Europe; you have DNS issues along the way to AVSIM.

I still get those 503 errors at those times, every day like clockwork, when accessing forum.avsim.net. I remember Tom mentioning he thought it was a European DNS issue, but aloso seem to remember it ultimately transpired that was when the forum server was running its daily backup, as Gartro has mentioned:

Sometimes, I would get a 5xx error message but other times I would give up waiting and come back later.

 

Because I was well aware the time in eastern USA (one of my sons and his family had been living in Pittsburgh PA), I has assumed that the Forum was "down" at that early morning time each day for maintenance or similar.

 

I think it ultimately boils down to the fact the forum servers are stretched.

[DarkstarF16 72]

It's unlikely that the servers are stretched during the times I've mentioned as I would imagine it's a low traffic period.

 

Because the UK is now on British Summer Time 9am translates to:

4am (0400hrs) on the east coast

1am (0100hrs) on the Pacific coast

6pm (1800hrs) in Australia

And during the working day or end of day for most of Europe Asia etc. the USA is mostly tucked up in bed.

Central America is just getting up and in For Africa it's early to mid morning.

 

The clockwork regularity points to a server job running somewhere. If I was running jobs, it's logical to set them to run during the period of least traffic when the majority of your users are asleep.

 

@codechris - your right, it's unlikely to be a DNS issue. I did some quick tests this morning during the down period:

Pinging forum.avsim.net [205.252.89.43] with 32 bytes of data:

Reply from 205.252.89.43: bytes=32 time=107ms TTL=57

Reply from 205.252.89.43: bytes=32 time=107ms TTL=57

Reply from 205.252.89.43: bytes=32 time=107ms TTL=57

Reply from 205.252.89.43: bytes=32 time=116ms TTL=57

Ping statistics for 205.252.89.43:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 107ms, Maximum = 116ms, Average = 109ms

 

Tracing route to forum.avsim.net [205.252.89.43]

over a maximum of 30 hops:

1 * 2 ms 2 ms 192.168.0.1

2 25 ms 21 ms 22 ms host-92-26-144-1.as13285.net [92.26.144.1]

3 24 ms 42 ms 20 ms host-78-151-227-8.as13285.net [78.151.227.8]

4 25 ms 21 ms 22 ms host-78-151-227-5.as13285.net [78.151.227.5]

5 29 ms 30 ms 28 ms host-78-144-10-173.as13285.net [78.144.10.173]

6 31 ms 37 ms 28 ms te7-5.br03.ldn01.pccwbtn.net [195.66.224.167]

7 109 ms 110 ms 111 ms purple.servers.avsim.net [205.252.89.43]

Trace complete.

 

Bri

[ckyliu 1,749]

It's unlikely that the servers are stretched during the times I've mentioned as I would imagine it's a low traffic period.

 

The clockwork regularity points to a server job running somewhere. If I was running jobs, it's logical to set them to run during the period of least traffic when the majority of your users are asleep.

That's what I meant? Unable to serve all requests whilst backing up at the same time is what I'd call stretched.

[DarkstarF16 72]

That's what I meant? Unable to serve all requests whilst backing up at the same time is what I'd call stretched.

I see, I misunderstood your post. Either way, you also cannot rule out someone running a DOS script. Either way, it should be traceable.

[codechris 9]

Backup script sounds likely. Most people run their automated jobs overnight. Could either be web server(s) or database servers(s) that are having a hard time processing requests. I would take an educated guess at the database server.

 

Best thing to do is backup what isn't live. So if there is a Active/passive database nodes, backup the passive. Same with web servers, take them out the load balancer pool, back them up, put them back in. I can't say I've ever had to backup a web server though, there is no need. Could be any other automated script though, not just backup. Should be easy to fix, maybe just the priority of the job needs to be lowered. Many ways, but none of them the most challenging in the world thankfully

[mgh 89]

I'm in the UK and also get the downtime every morning as regular as clockwork.

 

But, AVSIM is the only site that exhibits this behaviour. That's why I suspect this a  local AVSIM problem.

[jdwgraf 52]

http://forum.avsim.net/topic/435990-503-service-unavailable/#entry2959123

 

Above is the answer given by the man himself when I was unable to log in between 09:00 and 10:00 GMT ( and is still the case ) .

 

John

[vc10man 732]

 

 

I'm in the UK and also get the downtime every morning as regular as clockwork.

But, AVSIM is the only site that exhibits this behaviour

 

Ditto here too,in UK.