Recurly Application Unavailable
Incident Report for Recurly

Start Time: 11.11.15: 09:54 PST

Stop Time: 11:11:15: 10:07 PST

Update: During a routine maintenance update to our backend infrastructure there were parameters left over in our configuration management tool that were not properly cleaned up. Following the maintenance a low risk operating system patch was implemented causing the configuration management tool to pick up the leftover parameters which in turn impacted the production backend infrastructure. The operations staff was notified immediately and we resolved the situation with 11 minutes of downtime and an additional 2 minutes of degraded service.

Remediation: Moving forward, run books have been modified to enforce the clean up phases of maintenances and additional training for our operations staff.

We apologize for the interruption in service, and thank you for your patience during that time.

Regards,

Recurly Operations and Support Staff

Posted about 2 years ago. Nov 12, 2015 - 16:36 PST

Resolved
This morning's issue is resolved. The export and reporting services are slightly behind and should be caught up shortly.
Posted about 2 years ago. Nov 11, 2015 - 15:39 PST
Monitoring
The service is fully available. Background jobs are catching up.
Posted about 2 years ago. Nov 11, 2015 - 11:08 PST
Identified
We've identified the issue and the service will be coming back online
Posted about 2 years ago. Nov 11, 2015 - 10:09 PST
Investigating
The Recurly application is currently unavailable. Our ops team is investigating and we will have an update shortly.
Posted about 2 years ago. Nov 11, 2015 - 09:58 PST