Our engineering team has deployed the fix for the 360dialog Hub UI issue presented and now all systems are operational.
We will keep monitoring its behavior to ensure it is stable and will provide an update in the upcoming hours.
Root Cause Analysis
Issue description and impact:
On 11/04/23 at ~10:50 UTC we detected a disruption to our core services. Specifically, calls to our WhatsApp and 360dialog Hub APIs were timing out. Our web application, the 360dialog Hub, could not be loaded.
Cause of the issue:
A malfunction in our Redis based caching service caused by a bug in the software.
Mitigation Steps / Summary:
Our engineering team applied patches which mitigated the issue as per the summary below.
360dialog Hub - Partners webservice
5 hours and 24 minutes
10:50-16:14 UTC
360dialog Hub - Other services
3 hours and 7 minutes
10:50-13:57 UTC
Messaging Waba proxy callback and Waba proxy webserver
1 hour and 48 minutes
10:50-12:38 UTC
(partial outage)
Prevention of recurrence & suggestions:
We have completed an internal review with all relevant teams and work is underway to strengthen our deployment protocols to prevent recurrence.