Production Service Disruption — Database Storage

Duration: Approximately 4 hours of intermittent disruption (03:00 – 06:51 UTC), with two distinct impact windows.

What Happened

Pinpoint experienced an unexpected service disruption in the early hours of 13 May. The incident had two impact windows: an initial period of unavailability between approximately 03:00 and 04:00 UTC, and a follow-on period between approximately 05:39 and 06:51 UTC during which users attempting to modify certain types of records (predominantly scorecard submissions) encountered errors.

No data was lost or corrupted, and no background jobs were delayed.

Timeline (UTC)

• 03:00
  Initial impact begins; application unavailable.
• 04:00
  Initial impact resolves; engineering team investigating.
• 05:39
  Second impact window begins; errors on a subset of record updates.
• 06:29
  Status page warning posted.
• 06:51
  Second impact window resolves.
• 07:04
  Incident marked as resolved.

Why It Happened

The disruption was caused by an unexpected interaction between a routine internal database operation and an active background process. This combination of events is an outlier we had not previously encountered or anticipated. The root cause has been fully investigated and is understood.

What We're Doing About It

• Additional Monitoring
  We have put additional monitoring in place to detect this specific class of interaction much earlier, before it can cause customer impact.
• Increased Database Capacity
  We have provisioned additional database capacity to further reduce the likelihood of recurrence.

We sincerely apologise for the disruption and appreciate your patience.