This is an update to our preliminary blog post regarding our hardware failure.
This post provides more detail and insight into the current situation.
On Monday at 3:30am PDT, we experienced a hardware failure in our primary encryption hardware device. The failure cascaded to the backup slave device as well. This failure corrupted encryption keys used to access stored credit cards to process recurring transactions. At this point, it remains unclear how much of this data will be retrievable.
We have been working with our vendor and their many partners to receive their expert assessments of the situation. However, the particular failure is in a device that is designed specifically to make retrieval of information extremely difficult. In this situation, we have found ourselves at odds with the very protections we have worked so hard to put in place.
Current functionality of Recurly while we resolve this issue:
• New customer sign-ups continue uninterrupted
• All recurring invoice jobs are paused and queuing
• While transactions are paused, our live 'Production' customers will experience an interruption to their daily recurring collections
• Charges applied to any end-customer account will continue to be added to pending invoices
• The ability to process refunds may be impacted
Our plan going forward:
We are working to restore as much data as possible as quickly as is practicable. There will be three different outcomes for our customers.
1) Some of our customers have already been fully restored
2) Some customers will be restored to full functionality over the coming days as we continue to re-populate credit card data
3) Some customers will be required to reach out to (some or all) of their customers to have them re-enter billing information. If and when this is necessary, we will provide support and tools to make this easy.
We will continue to update our Production customers directly via email as we make further progress. We will be restoring our credit card data store and re-enabling recurring invoicing for our customers over the coming days.
Updates will be available via @recurlystatus, status.recurly.com, and this blog.