Hello,
www16.a2hosting.com has experienced an issue with it's drive subsystem. This error reverted all data to a point on approximately April 22nd. If your site is on this server you may have noticed that all of the data "time traveled" five days into the past.
We are restoring from the latest backup before the occurrence as of last night. Please note that this will take a few hours to perform, and the restore point will be from April 26th at approximately 10:20p.m.
If you have any questions regarding this, please feel free to file a support ticket, and our representatives will be happy to answer any questions.
We are still working on getting everybody access to their data. We realize everybody would like an explanation how this happened and why we are having trouble getting everything back to a stable state. Below is the root cause as we currently understand it.
We had a hard drive failure on the 22nd and while we use RAID technology so this shouldn't be a problem and we have done hard drive replacements hundreds of times, in this particular situation there seems to have been some problem and confusion within the RAID system software between the working and broken RAID disks and it seems to have continued writing data on the disk with errors instead of the working one. This is how the "time travel" occurred when we went to replace the disk this weekend because we figured out the disk was bad and the correct one actually was the one that stopped having data written to it on the 22nd. We had to revert back to the data from the 22nd because we couldn't trust the data on the other drive to be error free due to the problem. This in turn caused problems with our backup software because some sectors on the drive were bad on the backups created since the 22nd and that's why we were unable to perform a proper point in time restore like we had planned. We are working with our vendor on this problem and are investigating the root cause of all this and how we can avoid it in the future.
We are going to try to give everybody access to the backup snapshots so that you can go back and recover data that was backed up since the 22nd.
We will keep everybody updated as we make progress and of course apologize for this inconvenience. This is the first time we've seen a problem of this type occur, but rest assured we will be investigating how this happened exactly and what procedures we can use in the future to try to make sure this can't happen again.
Thank you and we appreciate all your understanding and patience with this matter. This situation is extremely frustrating for us as well, as we had systems and software in place to avoid this type of situation that was tested and obviously has come up less than expected in a real world scenario.
Customers on a2s16 should have received an email on how to access their backup data from the control panel cPanel and how to restore any data you may need that was on the backup since the 22nd.
If you continue to have problems with this, please submit a ticket to support and we will do our best to assist you.
Thank you for your understanding.