Test: 15/06/2021 02:00 BST - V2 SaaS:EU West - Database down due to Azure misconfiguration - Customer Affecting: Yes

Incident Report for Speechmatics

Postmortem

Azure human error

See full MIR Here https://speechmatics.atlassian.net/wiki/spaces/TS/pages/2035188021/2021+-+Major+Incident+Reports

Posted Jun 16, 2021 - 07:19 UTC

Resolved

04:00 resultion fix has restored the service to full availability.

Posted Jun 15, 2021 - 12:58 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jun 15, 2021 - 12:57 UTC

Identified

03:00 - Issue identified and support ticket raised with Microsoft.

Posted Jun 15, 2021 - 12:56 UTC

Investigating

Issue Summary: At 02:00 BST this morning due to human error, Microsoft Azure reconfigured the Postgress Database incorrectly causing it to become unavailable. This caused all customer the jobs currently submitted for processing to be rejected.

Customers Affected: Customer1, Customer2, Customer3

Issue Responsibilty: Microsoft Azure.

Resolution Responsibility: Microsoft Azure.

Current plan of Action or Workaround: A Severity 1 ticket has been raised with Microsoft Azure who have acknowledge it and are working on a resolution. No workaround available to Speechmatics.

Anticipated Time to Resolution: unknown.

Posted Jun 15, 2021 - 12:55 UTC

This incident affected: Batch SaaS - EU (Batch SaaS - EU1).