While websites were not “down” in a traditional sense, there was a period of about 30 minutes where users would experience the above error.

No data was lost.

• Plesk - to version 8.4.0 (what’s new)
• Apache (the web server) - to version 2.2.8-1.fc8
• PHP - to version 5.2.4-3
• MySQL - to version 5.0.45-6.fc8
• Ruby - to version 1.8.6.114-1.fc8
• SpamAssassin - to 3.2.4-1.fc8

Background

On 12 May 2008, we initiated a request to our datacenter to upgrade the operating system on our primary server (melvin.smilingpeanut.com) to Fedora Core 8. (We had previously been running FC4.) We were told this would be routine and that backups and transfers of data would be taken care of with minimal involvement on our part. That said, we made sure to back up core data such as user files, databases and mail on our own just in case. The update was to commence on 14 May 2008 in the very early hours of the morning so as to impact our customers as little as possible.

And we’re down

Shortly after midnight on 14 May, the server was taken offline to start the upgrade procedure. We were told this would take five hours at the most. We had also given instructions to the datacenter techs to restore data simultaneously so that the server would begin serving pages at the earliest second after the upgrade was completed.

About five hours later, we were informed by the datacenter that the server was back up and running.

Later that morning, we were alerted by customers that their files were missing. Upon investigation and communication with the datacenter, we discovered that customer data had not been restored because the server hard drive had been replaced with a fresh one, and all customer data was still on the original drive. This was not what we were told would happen.

We were then forced to wait for the original drive to be physically put back into the server and mounted so that we could recover user data. To compound matters, we were further informed that the settings backup for our account management software (Plesk) was corrupt. (We did not make our own backup of this information because we were told there was no need.)

Then we waited.

Why did it take so long?

Because the Plesk data was corrupt, we could not migrate user data from the other drive until that situation was fixed. To fix this, the datacenter once again removed the original drive (running FC4) and put it into another server to get a version of Plesk running so that the setting information could be exported. Once they had a proper setting backup, they reinserted the original drive back into the server and successfully migrated the Plesk settings on the new drive.

Sometime during this migration, Plesk (on the new drive) crashed and the MySQL database that Plesk depends on became corrupted. It took about three hours before Plesk became usable again and we could proceed with the migration.

The next several hours became a process of checking, rebooting, fixing permissions and running down issues. We restored service at approximately 2:25am MST. Later in the morning, we ran down some other issues that were discovered overnight. All known issues were rectified at approximately 9:00am MST.

No data was lost during this upgrade.

The timeline

All times MST.

So what now?

As a result of this extended outage, we are taking additional measures to make sure this never happens again:

1. Plesk user data will now be backed up on its own, separate from the Sunday night backups.
2. We will now coordinate with the datacenter over the phone during and after any important system upgrades.
3. We will add a second offsite backup location of both user data and Plesk settings. In the event of any extended maintenance, we will immediately switch DNS to the backup server in order to continue serving web pages.

In reviewing the last two days, we realized that in the event of something like this, we found it very difficult to update clients on what was going on. To fix this, we’re going to make use of the Twitter service, located at:

http://twitter.com/smilingpeanut

Twitter allows you to receive these notices via SMS text message, email and web. This is a free service.

Please bookmark this address in your browser. In the event of a loss of communications, we will post notices and updates there. We may dedicate an official offsite website for client communications at a later date.

The server has been running smoothly for nearly ten hours. We will continue to aggressively monitor the server’s health and stability to ensure no other problems creep up.

In closing

On behalf of everyone at SMILING PEANUT, the folks at the datacenter and everywhere in between, I once again apologize for the problems this has caused. We remain committed to learning from this and making sure it never happens again. If you would like to further discuss this, please call us at 970.449.0844.

Thank you, as always — but particularly today — for your support of SMILING PEANUT.

Chris Lanphear

UPDATE @ 0225 MST: We have verified that all websites hosted on Melvin are functional.  For those interested, we will post a timeline as well as a thorough explanation of the day’s events later today.  But for now, we need some rest.

We are working with the datacenter to minimize downtime as much as possible.  No data will be lost while the server is offline, and we are working to make sure that data and connectivity will be restored as soon as humanly possible.

We have three redundant backups in case of any unexpected problems, and once the server is back online, it will immediately begin serving web sites and mail as normal.

Please check smilingpeanut.net for updates as this maintenance gets underway.  Should you have any additional questions, please contact us at 970.449.0844.

Thank you for your continued support of SMILING PEANUT web hosting.  We sincerely appreciate your business.

From all of us at SMILING PEANUT, we wish you and yours a happy and prosperous holiday season.

Upon reboot, all services have been returned to normal.  No data loss was reported.

No downtime was needed for this upgrade. 

Upon reboot, all services have been returned to normal.  No data loss was reported.

After some investigation, it has been determined that the downtime is not due to a problem with the server, and no other sites have been affected. We have already taken the necessary actions to correct this problem, although we currently do not have an estimated time of resolution.

We will update this post as things develop.

In the meantime, you may access the site by using techops.smilingpeanut.com.

UPDATE 3:14pm MST: Our DNS server shows that techops.net is back. Depending on your ISP, you may see this change immediately or sometime within the next 72 hours. This is not under our control. We will continue to maintain the mirror URL above for those of you whose DNS has not refreshed.