Date: January 5th, 2015
Time: 11:02 thru 11:28am PST
On January 5th, 2015 Recurly Inc.’s production network experienced an outage lasting from 11:02am thru 11:28am PST. The root cause analysis of the incident concluded that during a previous routine network maintenance, a network configuration parameter was not adequately set, thus triggering an escalation event culminating in the outage. Once the issue was identified, immediate steps were taken to remediate the issues and restore production connectivity.
Once the initial incident response was complete, an internal postmortem was held on Tuesday January 6th 2015. Additional controls have now been put in place for the Network Engineering team to ensure requirements and limitations surrounding configuration parameters are reviewed and tested to ensure connectivity.