360dialog - Application request limit reached – Incident details

All systems operational

Application request limit reached

Resolved
Degraded performance
Started about 1 month agoLasted 5 days

Affected

360dialog Infrastructure

Degraded performance from 6:43 PM to 8:25 AM, Operational from 6:43 PM to 8:25 AM, Degraded performance from 8:25 AM to 10:47 AM, Operational from 8:25 AM to 10:47 AM, Degraded performance from 10:47 AM to 3:21 PM, Operational from 10:47 AM to 3:21 PM, Degraded performance from 3:21 PM to 8:09 AM, Operational from 3:21 PM to 8:09 AM, Degraded performance from 8:09 AM to 3:33 PM, Operational from 8:09 AM to 3:33 PM, Degraded performance from 3:33 PM to 5:03 PM, Operational from 3:33 PM to 5:03 PM, Degraded performance from 5:03 PM to 6:35 PM, Operational from 5:03 PM to 7:24 AM

360dialog WhatsApp Business API

Degraded performance from 6:43 PM to 6:35 PM, Operational from 6:35 PM to 7:24 AM

WhatsApp Business API Number Setup

Operational from 6:43 PM to 7:24 AM

360dialog Client Hub

Degraded performance from 6:43 PM to 6:35 PM, Operational from 6:35 PM to 7:24 AM

360dialog Partner Hub

Degraded performance from 6:43 PM to 6:35 PM, Operational from 6:35 PM to 7:24 AM

Partner API

Degraded performance from 6:43 PM to 6:35 PM, Operational from 6:35 PM to 7:24 AM

Updates
  • Resolved
    Resolved

    The issue has been mitigated with no re-occurrence since 14:48 GMT+1 on 16/09/24.
    ---
    We will attach an RCA here we have completed a full investigation with Meta.

  • Monitoring
    Update

    Affected partner users are unblocked. The issue is resolved.

    We will provide an RCA here in due course when we complete a review of the incident with Meta.

  • Monitoring
    Monitoring

    API requests should no longer receive rate limit errors.
    WABA management operations have been restored.
    We are currently restoring access to partner user accounts that were temporarily blocked.

  • Identified
    Update

    To safeguard overall system performance, we were required to take immediate action and temporarily block some Partner User accounts from which high volume of API requests being made. We will unblock the affected accounts as soon as possible when we validate that the traffic as authentic.

  • Identified
    Update

    Our engineering and support teams are continuing to work on resolving this issue.
    In the meantime, we recommend using Meta Business Manager to manage templates and WABA profiles.

    We appreciate your patience during this time, and we will continue to provide updates as soon as we have more relevant information.

  • Identified
    Update

    Our engineering team is actively working on resolving this issue, and we remain in constant communication with Meta Support to expedite a solution. While we do not yet have an estimated time for resolution, please know that we are doing everything possible to address the situation swiftly.

    This issue is impacting several key features, including migrations, onboarding, templates, flows, hosting type change, and profile information. As a temporary measure, we will be disabling the sync worker for templates to help mitigate these challenges.

    In the meantime, we recommend using Meta Business Manager as a workaround for creating and submitting templates. Additionally, we advise refraining from performing migrations, onboarding, and template-related actions using the 36dialog Hub or API for now. This will help prevent the limit from being reached immediately once the issue is resolved and ensure a smoother experience when the systems are fully operational again.

    We greatly appreciate your patience and understanding during this time, and we will continue to provide updates as soon as we have more information.

  • Identified
    Update

    There is also a rate limit for editing and submitting templates or making API calls.

    Initiating more requests will result in more errors.

    If you receive errors while submitting the template or receive HTTP 400 response from API call, try lowering the number of requests.

  • Identified
    Update

    Editing or submitting templates from the HUB and API still produce errors. This applies both to partner and client accounts. We are continuing to work on this problem.

  • Identified
    Identified

    We are aware of an issue impacting some activities on the 360dialog Hub and API. Our engineering team is currently investigating it. We do not currently have a resolution ETA but we'll update again as soon as possible.