360dialog Infrastructure
Operational
360dialog WhatsApp Business API
Operational
WhatsApp Business API Number Setup
Operational
360 Client and Partner Hub
Operational

Partner API

Operational
Partner Webhooks
Operational

Marketing Automation

Operational
Meta / Facebook Infrastructure
Operational
Meta / Facebook WhatsApp Business API (Message Delivery)
Operational

WhatsApp Business Account Management (FB Business Manager)

Operational
Embedded Signup
Operational

Notice history

Apr 2023

Instability after upgrade to WABA version 2.47.1
Some disruptions to Cloud API messaging, Embedded Signup flow and other services
  • Resolved

    The issue is now resolved.

  • Monitoring

    The issue affecting Cloud API message delivery has been mitigated. We are monitoring and keep track of the issue to make sure it is fully resolved.

  • Monitoring

    The issue affecting Cloud API message delivery has been mitigated. We are containing to monitor the situation.

  • Identified

    We are working with Meta's Integration Support Engineering team to restore Cloud API messaging. Cloud API numbers are still currently unable to receive messages.
    We will provide another update within 4 hours or sooner if additional information is available.

  • Identified

    There is an ongoing issue that preventing Cloud API numbers from receiving messages. We are working on a solution currently. ETA: ~1 hour

  • Resolved

    The issue is now resolved.

  • Identified

    We have identified the cause of the issue. We are working with Meta's Integration Support Engineering team to restore the services.

  • Investigating

    We are aware of an issue that is impacting some of our platform components. Specifically, this issue is effecting:

    • Cloud API messaging
    • Embedded Signup flow (and integrated onboarding flow)
    • Template management
    • Profile management
    • WhatsApp webhooks
    • Migrations (number porting)
    • On-premise -> Cloud API migration

    360dialog On-Premise messaging is not affected.

    Our engineering team is currently investigating the issue with Meta and we'll update again as soon as possible.

Some disruptions to WhatsApp API and 360dialog Hub
  • Resolved

    Our engineering team has deployed the fix for the 360dialog Hub UI issue presented and now all systems are operational.

    We will keep monitoring its behavior to ensure it is stable and will provide an update in the upcoming hours.

    Root Cause Analysis

    Issue description and impact:
    On 11/04/23 at ~10:50 UTC we detected a disruption to our core services. Specifically, calls to our WhatsApp and 360dialog Hub APIs were timing out. Our web application, the 360dialog Hub, could not be loaded.

    Cause of the issue:
    A malfunction in our Redis based caching service caused by a bug in the software.

    Mitigation Steps / Summary:
    Our engineering team applied patches which mitigated the issue as per the summary below.

    360dialog Hub - Partners webservice
    5 hours and 24 minutes
    10:50-16:14 UTC

    360dialog Hub - Other services
    3 hours and 7 minutes
    10:50-13:57 UTC

    Messaging Waba proxy callback and Waba proxy webserver
    1 hour and 48 minutes
    10:50-12:38 UTC
    (partial outage)

    Prevention of recurrence & suggestions:
    We have completed an internal review with all relevant teams and work is underway to strengthen our deployment protocols to prevent recurrence.

  • Identified

    We are aware that the 360dialog Hub UI is still inaccessible for many Customers. Our engineering teams are working on a solution for this. We will provide another update within 2 hours or sooner if additional information is available.

  • Identified

    We are aware that the 360dialog Hub UI is still inaccessible for many Customers. Our engineering teams are working on a solution for this. We will provide another update within 2 hours or sooner if additional information is available.

  • Identified

    The WhatsApp API has been restored ✅
    We are working to resolve the issues with the 360dialog Hub currently.

  • Identified

    Our engineers are working on a solution for the outage. We will update again asap.

  • Identified

    Our engineers have identified the cause of the issue.

  • Investigating

    We are aware of an issue that is impacting our WhatsApp API and 360dialog Hub. Our engineering team is currently investigating it and we'll update again as soon as possible.

    When attempting to send messages, you will get the error: 500 Internal Server Error
    When trying to load the 360dialog Hub or calling any hub.360dialog.com/v1 endpoint, you will get the error: 504 Gateway Time-out

Mar 2023

Mar 2023 to May 2023