Meta have applied an exception to our app.
(The exception was incorrectly applied on Fri 27/09/24 but the issue was fixed by Meta's engineering team). We do not expect the issue to re-occur.

360dialog - Root Cause Analysis

Issue description and impact:

On 12th Sept 2024 ~14:45 utc+1, waba management operations were blocked:

Template creation
Template syncing
Number registration
WABA profile management
Migrations
Onboarding
Channel updates

Cause of the issue:

The issue arose due to Meta setting the rate limit for one of our applications too low. This threshold, controlled and enforced at the application level by Meta, led to a disruption in our service. This happened because 360dialog has significantly more active WABAs than any other BSP and Meta has failed to apply proper limits according to our strong growth. While we strive to maintain optimal performance, the constraints set by Meta directly impacted the functionality of our application in this instance.

Mitigation Steps:

Sept 12, 14:45 - Issue started

Sept 13, 07:46 - Issue escalated to Meta

Sept 13, 15:20 - Attempted fix

No response from Meta

Sept 14, 13:32 - Initiated workaround

Sept 14, 15:40 - App unblocked. Issue mitigated temporarily.

—

Sept 16, 06:51 - Issue re-occurs

Sept 16, 13:30 - Meta provides support that results in root cause diagnosis

Sept 16, 14:46 - Issue mitigated temporarily

—

Sept 25, 12:55 - Issue re-occurred

Sept 27, 13:30 - Workaround measure deployed

Sept 27, 15:10 - Issue mitigated temporarily

Sept 27, 17:00 - Meta confirm they applied an exception to our app

–

Sept 30, 14:23 - Issue reoccurs

Sept 30, 16:53 - Meta confirmed the exception was not applied correctly on their side but they fixed it.

Issue Resolved

Prevention of recurrence & suggestions:

We have obtained an adequate rate limit again and are working with Meta on a process to increase our limit accordingly to our growth in the future. Additionally we have completed an internal review with relevant teams & stakeholders and we’re working on adjusting our system architecture to accommodate Meta’s app rate limiting practices.

Separately, Meta is conducting a thorough investigation to understand where the escalation process failed and how they can improve it, particularly during weekends.