- - Some disruptions to WhatsApp API and 360dialog Hub – Incident details

Our engineering team has deployed the fix for the 360dialog Hub UI issue presented and now all systems are operational.

We will keep monitoring its behavior to ensure it is stable and will provide an update in the upcoming hours.

Root Cause Analysis

Issue description and impact:
On 11/04/23 at ~10:50 UTC we detected a disruption to our core services. Specifically, calls to our WhatsApp and 360dialog Hub APIs were timing out. Our web application, the 360dialog Hub, could not be loaded.

Cause of the issue:
A malfunction in our Redis based caching service caused by a bug in the software.

Mitigation Steps / Summary:
Our engineering team applied patches which mitigated the issue as per the summary below.

360dialog Hub - Partners webservice
5 hours and 24 minutes
10:50-16:14 UTC

360dialog Hub - Other services
3 hours and 7 minutes
10:50-13:57 UTC

Messaging Waba proxy callback and Waba proxy webserver
1 hour and 48 minutes
10:50-12:38 UTC
(partial outage)

Prevention of recurrence & suggestions:
We have completed an internal review with all relevant teams and work is underway to strengthen our deployment protocols to prevent recurrence.

We are aware that the 360dialog Hub UI is still inaccessible for many Customers. Our engineering teams are working on a solution for this. We will provide another update within 2 hours or sooner if additional information is available.

We are aware that the 360dialog Hub UI is still inaccessible for many Customers. Our engineering teams are working on a solution for this. We will provide another update within 2 hours or sooner if additional information is available.

The WhatsApp API has been restored ✅
We are working to resolve the issues with the 360dialog Hub currently.

Our engineers are working on a solution for the outage. We will update again asap.

Our engineers have identified the cause of the issue.