360Dialog - Notice history

Meta have applied an exception to our app.
(The exception was incorrectly applied on Fri 27/09/24 but the issue was fixed by Meta's engineering team). We do not expect the issue to re-occur.

360dialog - Root Cause Analysis 

Issue description and impact:

On 12th Sept 2024  ~14:45 utc+1, waba management operations were blocked: 

• Template creation

• Template syncing

• Number registration

• WABA profile management

• Migrations

• Onboarding

• Channel updates

Cause of the issue:

The issue arose due to Meta setting the rate limit for one of our applications too low. This threshold, controlled and enforced at the application level by Meta, led to a disruption in our service. This happened because 360dialog has significantly more active WABAs than any other BSP and Meta has failed to apply proper limits according to our strong growth. While we strive to maintain optimal performance, the constraints set by Meta directly impacted the functionality of our application in this instance.

Mitigation Steps:

Sept 12, 14:45 - Issue started

Sept 13, 07:46 - Issue escalated to Meta

Sept 13, 15:20 - Attempted fix

No response from Meta

Sept 14, 13:32 - Initiated workaround

Sept 14, 15:40 - App unblocked. Issue mitigated temporarily.

—

Sept 16, 06:51 - Issue re-occurs

Sept 16, 13:30 - Meta provides support that results in root cause diagnosis

Sept 16, 14:46 - Issue mitigated temporarily

—

Sept 25, 12:55 - Issue re-occurred

Sept 27, 13:30 - Workaround measure deployed

Sept 27, 15:10 - Issue mitigated temporarily

Sept 27, 17:00 - Meta confirm they applied an exception to our app

–

Sept 30, 14:23 - Issue reoccurs

Sept 30, 16:53 - Meta confirmed the exception was not applied correctly on their side but they fixed it. 

Issue Resolved

Prevention of recurrence & suggestions:

We have obtained an adequate rate limit again and are working with Meta on a process to increase our limit accordingly to our growth in the future. Additionally we have completed an internal review with relevant teams & stakeholders and we’re working on adjusting our system architecture to accommodate Meta’s app rate limiting practices. 

Separately, Meta is conducting a thorough investigation to understand where the escalation process failed and how they can improve it, particularly during weekends. 

Template message creation and profile photo upload functionality has been restored.

We are continuing to monitor the situation with Meta support.

We are aware that the issue is re-occurring and is affecting some WABA operations including:

• Profile photo upload

• Template message creation with header_handle and/or header_url

We are in contact with Meta about the issue and the case is escalated with their relevant stakeholders.

We will provide another update in 4 hours or sooner if we have a significant development.

Template message creation and profile photo upload functionality has been restored.

We are continuing to monitor the situation with Meta support.

We are continuing to work with Meta Engineering team on this issue.

The below actions are still affected:

• Cloud API profile photo upload

• Template message creation with header_handle and/or header_url

We will provide another update regards the status.

Our Engineering team is still working on a fix to make the below actions available:

• Cloud API profile photo upload

• Template message creation with header_handle and/or header_url

We will provide another update regards the status.

We are continuing to work on the below as this actions are still affected:

• Cloud API profile photo upload

• Template message creation with header_handle and/or header_url

We will share another update when this is resolved.

The majority of functionality has been restored to normal.

However, the below actions are still affected and we are working to restore full functionality

• Cloud API profile photo upload

• Template message creation with header_handle and/or header_url

We will share another update when this is resolved.

We are continuing to work on this problem, and do not currently have an ETA for restoration of all functionality.

The issue is now resolved.

We are pleased to inform you that the embedded sign-up issue on Meta infrastructure has been successfully resolved. The META engineering team have implemented the necessary fixes, and the system is now functioning as expected.

We will continue to closely monitor the situation to ensure stability and address any potential concerns promptly.

The issue has been mitigated with no re-occurrence since 14:48 GMT+1 on 16/09/2024.
--

We will attach an RCA here once we have completed a full investigation with Meta.

We are continuing to work on this problem with Meta engineering support. We are currently working on resolving the issue which is our top priority right now. We currently don't have an ETA for restoration of all functionality.

The issue has been mitigated with no re-occurrence since 14:48 GMT+1 on 16/09/24.
---
We will attach an RCA here we have completed a full investigation with Meta.

Affected partner users are unblocked. The issue is resolved.

We will provide an RCA here in due course when we complete a review of the incident with Meta.

API requests should no longer receive rate limit errors.
WABA management operations have been restored.
We are currently restoring access to partner user accounts that were temporarily blocked.

To safeguard overall system performance, we were required to take immediate action and temporarily block some Partner User accounts from which high volume of API requests being made. We will unblock the affected accounts as soon as possible when we validate that the traffic as authentic.

Our engineering and support teams are continuing to work on resolving this issue.
In the meantime, we recommend using Meta Business Manager to manage templates and WABA profiles.

We appreciate your patience during this time, and we will continue to provide updates as soon as we have more relevant information.

Our engineering team is actively working on resolving this issue, and we remain in constant communication with Meta Support to expedite a solution. While we do not yet have an estimated time for resolution, please know that we are doing everything possible to address the situation swiftly.

This issue is impacting several key features, including migrations, onboarding, templates, flows, hosting type change, and profile information. As a temporary measure, we will be disabling the sync worker for templates to help mitigate these challenges.

In the meantime, we recommend using Meta Business Manager as a workaround for creating and submitting templates. Additionally, we advise refraining from performing migrations, onboarding, and template-related actions using the 36dialog Hub or API for now. This will help prevent the limit from being reached immediately once the issue is resolved and ensure a smoother experience when the systems are fully operational again.

We greatly appreciate your patience and understanding during this time, and we will continue to provide updates as soon as we have more information.

There is also a rate limit for editing and submitting templates or making API calls.

Initiating more requests will result in more errors.

If you receive errors while submitting the template or receive HTTP 400 response from API call, try lowering the number of requests.

Editing or submitting templates from the HUB and API still produce errors. This applies both to partner and client accounts. We are continuing to work on this problem.