Etsy Crashes, Suffers Data Loss

[amy]

Hi all,

As you can see from looking around, we're missing all data from the last day and a half. The database server had a hardware failure and we had to restore from a backup. We've sent out an email to all the sellers with an explanation, which I'll paste below.

We're all in tears over this, and we need your help to put things back in their place. Please contact your customers and let them know what happened, and we'll be here to help with everything we can.

blog updates: http://www.etsy.com/forums_topic.php?thread_id=1552

email to sellers:

Hi Etsy sellers,

At 3am on Saturday, February 11th, Etsy's database server suffered a hardware failure. We ran to the data center and were able to fix the computer, but we had to restore a backup of the database.

This backup of the database is about 30 hours old. So Etsy has a bit of memory loss. Everything that happened on Etsy in the past 30 hours has been forgotten: items listed, items sold, items edited, forum posts, conversations.

We feel awful about this, and we're going to do whatever it takes to make it up to you.

How?

*Member Indulgence Weekend*

Etsy is nothing without its members. This weekend, Etsy is *free* to use. To make up for losing 30 hours of data we're losing our fees for more hours than that. Listings are free and sales are free.

Customer support will be 'round the clock, on the forums and via email. You'll get quick replies to all questions.

We can also make Member Indulgence Weekends a tradition, like Ben & Jerry's Free Cone Day.

*What are we doing to make sure this doesn't happen again?*

We've just put in an order to get a full 5 cabinets of hardware space in the new datacenter. We'll make a big investment in new servers and have redundant redundancy. We're going to make backups more frequently and have entire servers whose only job is to run mirrored copies of the live database.

*Please Help*

If you bought, sold or listed an item on Etsy in the past day and a half, you'll have to re-buy / re-sell /re-list the item. If we work together we can get restore a lot of what was lost. We'll keep forfeiting sleep and the rest of our lives until the new hardware is in place and greater redundancy secured.

We have sent this email to all the sellers on Etsy. We did this because we didn't want to send it to thousands of people who are not affected by the data recovery (buyers who have not bought anything in the past day and half). Feel free to forward it to any of your customers, and have them contact us with any questions.

The new hardware will be installed as quickly as possible, and we'll use our blog to post updates on its progress.

[xppman]

Now that's CS. I got an e mail about it.
Many others would try and hide it and deny it.

One BIG company comes to mind. 

[amy]

[quote author=xppman link=topic=2600.msg9235#msg9235 date=1139686355]
Now that's CS. I got an e mail about it.
Many others would try and hide it and deny it.

One BIG company comes to mind. 

[/quote]

Who?

Not Endicia.  They always notify customers when there is an outage and give refunds

Not Overstock.  They notify and have given refunds--notice I didn't say they can fix the problem, but they do notify customers.

Not Linkpoint or Authorize.net.  They both notify customers although neither one gives refunds.

Not RoadRunner.  They deny there is a problem but they aren't trying to hide anything.  Their CS Reps are clueless about anything but saying "unplug your modem and plug it back in" and wouldn't know what a system outage was if it hit them in the face.

I give up.  Who else? --> 

[xppman]

   

   

[bargainbloodhound]

This backup of the database is about 30 hours old.

Why?  :  Hourly backups might be a good idea.