At approximately 3:30 a.m. (PDT) we experienced an intermittent hardware failure, which prevented some transactions from processing. Our team was alerted and mobilized quickly.
We had both failover and extra replacement boxes on hand. Working with our vendor we were able to restore the service and re-rack several replacement boxes. In order to do this, we paused our recurring transaction jobs. This means that new customer sign-ups are not impacted, but customers may see a delay in the posting of recurring daily invoices.
We are running data migrations and working with several vendors tomorrow to restore the recurring jobs on our service. We will provide a lengthier update on the issue after our teams have resolved the issues and the service is restored completely.
We apologize for the impact and inconvenience to our customers' businesses.
We will be updating again as we have more clarity on full time to complete resolution.
Updates will continue to be posted to @recurlystatus on Twitter.