360dialog - Notice history

The issue is now resolved.

We are currently in the testing phase of our latest deployment, focusing on ensuring that the OTP verification status is correctly handled for the affected channels during on-premise to cloud API migrations. Once testing is complete and we successfully deploy the changes in the hub, we will provide you with an update.
Thank you for your continued patience as we work to resolve this incident. If you have any questions, please reach out to our support team if you have any immediate concerns.

We are still working on releasing the product update in response to recent changes from Meta.

For any concerns, please reach out to our customer support team. We appreciate your patience and we will keep you updated as more information becomes available.

Please note that, as previously mentioned, we are preparing a product update in response to recent changes from Meta, scheduled for release on Monday, October 21, 2024. Our engineering team is focused on implementing a solution.

For any concerns, please reach out to our customer support team. We appreciate your patience and we will keep you updated as more information becomes available.

We have prepared a product change in response to the changes on the Meta side. We plan to release these changes on Monday 21.10.2024. In the meantime, in case of any issues, please contact our customer support.

We are still actively working with our engineering team to find a solution for the recent changes related to OTP verification during on-premise to cloud API migrations. We will provide new updates as soon as we have more details.

We are continuing to work closely with our engineering team to address the recent changes affecting OTP verification during on-premise to cloud API migrations.

We appreciate your patience and understanding during this time. We will provide new updates as soon as we have more details to share. Please continue to check our status page for the latest information regarding this issue.

We are still actively working with our engineering team to find a solution for the recent changes related to OTP verification during on-premise to cloud API migrations. We will provide new updates as soon as we have more details.

We´re still working on a way to better handle this new change on our side.

Meta has identified and resolved a security issue on their side. This update, which was implemented without public announcement, has affected on-premise to cloud API migrations, requiring OTP (One-Time Password) verification.

We are currently working on implementing a solution within our product allowing OTP verification to be handled seamlessly during migrations. This will ensure that the transition process continues without disruption.

We will regularly post updates on our status page regarding this issue.

The issue is now resolved.

The engineering team is still working on the issue.

We'll provide further updates as soon as possible.

We’re still working on the fix. Our engineering team is continuing to make progress on resolving the issue. We will provide further updates as soon as we have more information.

Thank you for your patience.

We´re still working on the fix.

We are continuing to work on this problem.

We expect to release the fix on 10.10.2024 during the day.

We are continuing to work on this problem.

We expect to release the fix on 10.10.2024 during the day.

The engineering team is still working on the issue and making progress toward a fix. We'll provide further updates as soon as possible.

The development team has identified the issue and they´re working on a fix.

The issue has been mitigated.

Meta have applied an exception to our app.
(The exception was incorrectly applied on Fri 27/09/24 but the issue was fixed by Meta's engineering team). We do not expect the issue to re-occur.

360dialog - Root Cause Analysis 

Issue description and impact:

On 12th Sept 2024  ~14:45 utc+1, waba management operations were blocked: 

• Template creation

• Template syncing

• Number registration

• WABA profile management

• Migrations

• Onboarding

• Channel updates

Cause of the issue:

The issue arose due to Meta setting the rate limit for one of our applications too low. This threshold, controlled and enforced at the application level by Meta, led to a disruption in our service. This happened because 360dialog has significantly more active WABAs than any other BSP and Meta has failed to apply proper limits according to our strong growth. While we strive to maintain optimal performance, the constraints set by Meta directly impacted the functionality of our application in this instance.

Mitigation Steps:

Sept 12, 14:45 - Issue started

Sept 13, 07:46 - Issue escalated to Meta

Sept 13, 15:20 - Attempted fix

No response from Meta

Sept 14, 13:32 - Initiated workaround

Sept 14, 15:40 - App unblocked. Issue mitigated temporarily.

—

Sept 16, 06:51 - Issue re-occurs

Sept 16, 13:30 - Meta provides support that results in root cause diagnosis

Sept 16, 14:46 - Issue mitigated temporarily

—

Sept 25, 12:55 - Issue re-occurred

Sept 27, 13:30 - Workaround measure deployed

Sept 27, 15:10 - Issue mitigated temporarily

Sept 27, 17:00 - Meta confirm they applied an exception to our app

–

Sept 30, 14:23 - Issue reoccurs

Sept 30, 16:53 - Meta confirmed the exception was not applied correctly on their side but they fixed it. 

Issue Resolved

Prevention of recurrence & suggestions:

We have obtained an adequate rate limit again and are working with Meta on a process to increase our limit accordingly to our growth in the future. Additionally we have completed an internal review with relevant teams & stakeholders and we’re working on adjusting our system architecture to accommodate Meta’s app rate limiting practices. 

Separately, Meta is conducting a thorough investigation to understand where the escalation process failed and how they can improve it, particularly during weekends. 

Template message creation and profile photo upload functionality has been restored.

We are continuing to monitor the situation with Meta support.

We are aware that the issue is re-occurring and is affecting some WABA operations including:

• Profile photo upload

• Template message creation with header_handle and/or header_url

We are in contact with Meta about the issue and the case is escalated with their relevant stakeholders.

We will provide another update in 4 hours or sooner if we have a significant development.

Template message creation and profile photo upload functionality has been restored.

We are continuing to monitor the situation with Meta support.

We are continuing to work with Meta Engineering team on this issue.

The below actions are still affected:

• Cloud API profile photo upload

• Template message creation with header_handle and/or header_url

We will provide another update regards the status.

Our Engineering team is still working on a fix to make the below actions available:

• Cloud API profile photo upload

• Template message creation with header_handle and/or header_url

We will provide another update regards the status.

We are continuing to work on the below as this actions are still affected:

• Cloud API profile photo upload

• Template message creation with header_handle and/or header_url

We will share another update when this is resolved.

The majority of functionality has been restored to normal.

However, the below actions are still affected and we are working to restore full functionality

• Cloud API profile photo upload

• Template message creation with header_handle and/or header_url

We will share another update when this is resolved.

We are continuing to work on this problem, and do not currently have an ETA for restoration of all functionality.

The issue is now resolved.

We are pleased to inform you that the embedded sign-up issue on Meta infrastructure has been successfully resolved. The META engineering team have implemented the necessary fixes, and the system is now functioning as expected.

We will continue to closely monitor the situation to ensure stability and address any potential concerns promptly.

The issue has been mitigated with no re-occurrence since 14:48 GMT+1 on 16/09/2024.
--

We will attach an RCA here once we have completed a full investigation with Meta.

We are continuing to work on this problem with Meta engineering support. We are currently working on resolving the issue which is our top priority right now. We currently don't have an ETA for restoration of all functionality.

The issue has been mitigated with no re-occurrence since 14:48 GMT+1 on 16/09/24.
---
We will attach an RCA here we have completed a full investigation with Meta.

Affected partner users are unblocked. The issue is resolved.

We will provide an RCA here in due course when we complete a review of the incident with Meta.

API requests should no longer receive rate limit errors.
WABA management operations have been restored.
We are currently restoring access to partner user accounts that were temporarily blocked.

To safeguard overall system performance, we were required to take immediate action and temporarily block some Partner User accounts from which high volume of API requests being made. We will unblock the affected accounts as soon as possible when we validate that the traffic as authentic.

Our engineering and support teams are continuing to work on resolving this issue.
In the meantime, we recommend using Meta Business Manager to manage templates and WABA profiles.

We appreciate your patience during this time, and we will continue to provide updates as soon as we have more relevant information.

Our engineering team is actively working on resolving this issue, and we remain in constant communication with Meta Support to expedite a solution. While we do not yet have an estimated time for resolution, please know that we are doing everything possible to address the situation swiftly.

This issue is impacting several key features, including migrations, onboarding, templates, flows, hosting type change, and profile information. As a temporary measure, we will be disabling the sync worker for templates to help mitigate these challenges.

In the meantime, we recommend using Meta Business Manager as a workaround for creating and submitting templates. Additionally, we advise refraining from performing migrations, onboarding, and template-related actions using the 36dialog Hub or API for now. This will help prevent the limit from being reached immediately once the issue is resolved and ensure a smoother experience when the systems are fully operational again.

We greatly appreciate your patience and understanding during this time, and we will continue to provide updates as soon as we have more information.

There is also a rate limit for editing and submitting templates or making API calls.

Initiating more requests will result in more errors.

If you receive errors while submitting the template or receive HTTP 400 response from API call, try lowering the number of requests.

Editing or submitting templates from the HUB and API still produce errors. This applies both to partner and client accounts. We are continuing to work on this problem.