We are now fairly confident in understanding why we were getting 503 errors and which were driving everyone nuts over the last 3 months or so. We originally attributed the 503's to an architecture problem that had to do with they way we communicate with our MySQL database (which is where all the forum information is stored). We now know that it was something else.

 

I am not an I.T. guy, but will try to explain this in terms I do understand. In a RAID array of disks, if one disk gets corrupted, the corruption propagates to other disks in the array over time. This happens because RAID arrays are designed to replicate data between drives. If a single disk fails due to its degree of corruption, the other disks may or may not work as expected in a RAID configuration because of the replication of that corrupted data. What happened over the last week is exactly a result of this.

 

The 503's we believe now resulted from the increasing corruption of the drives and the time it took for the drives to fetch information and data from the increasingly corrupted disks. When disk 1 failed, the RAID array could not recover due to the state of corruption of the other disks. Because we were looking elsewhere (architecture) and we were assured that our RAID would work as advertised, we under estimated the impact of a disk failure. Had the system continued to work, we would have completed our (misguided) architecture review and changed out the drive. Obviously, the system had other thoughts.

 

The bottom line? I am pretty sure that given the performance of the system today, that the 503's are an issue of the past (unless caused by me in system updates / changes) and that when we get all new replacement drives in place plus two expansion drives installed, you will see outstanding performance.

OK, sounds hopeful ;o)  Good to have Avsim back up and running.  

 

Thanks!

 

Noel