Learner Community Unscheduled Downtime - 12/15

As many of you know by now, Learner Community had an outage yesterday. I wanted to walk you through the chain of events that led to the downtime.

Yesterday around 2 PM EST, we were working behind the scenes at the database level on behalf of a client. We created a few webpages and published their sites.

Around 3:30 PM EST we realized that publishing was getting stuck as one of the asset drives was reaching its capacity. Xerox IT was able to expand the drive to allow publishing to continue.

At 4:53 PM EST Learner Community went down due to an internal error. I and my team immediately began looking into the issue. Around 5:40 PM EST we were able to find the culprit.

• When we were working on those webpages earlier on behalf of a client, we created multiple pages with the exact same path/url (at the database level).
  • Learner Community will prevent you from doing this through the application’s UI, but because we were going behind the scenes to do this work, those same constraints didn’t protect us here.
• At 6:00 PM EST, we identified the pages that were causing the problem and deleted them. Learner Community began functioning again.

This could have been prevented if we had a simple unique database constraint to only allow one path/url per client. We’ve added the database constraint to prevent this issue from occurring again.

There is never a good time to be down, and I know many of your learners are working to get last minute things done. This is the first time that I can recall having unscheduled downtime caused by Learner Community itself. We deeply apologize for any issues this may have caused your organization. As described above, we’ve put steps in place to prevent this issue from occurring again.

We wanted to share the chain of events with you and hope it can bring you some comfort that we will always be open when it comes to any issues with Learner Community.