360Dialog - Notice history

All systems operational

Operational

360dialog WhatsApp Business API - Operational

WhatsApp Business API Number Setup - Operational

360dialog Client Hub - Operational

360dialog Partner Hub - Operational

Partner API - Operational

Partner Webhooks - Operational

Marketing Automation - Operational

360dialog Payments System - Operational

360dialog Sandbox - Operational

Operational

WhatsApp On-Premise API (Message Delivery) - Operational

WhatsApp Cloud API (Message Delivery) - Operational

WhatsApp Business Account Management (Meta Business Manager) - Operational

Embedded Signup - Operational

Notice history

May 2022

Problems with messaging
  • Resolved
    Resolved

    The issue is now resolved for the vast majority of setups.

    We have monitoring in place to proactively fix the issue in the rare cases it still happens.

    If you experience a problem related to this incident, please create a Support Ticket with us (Category: Messaging Has Completely Stopped Working) and we will assist you.

  • Identified
    Identified

    We are seeing that a small percentage of stacks are still having issues related to the incident reported by Meta on Tuesday.

    The clients affected by the issue get errors similar to {"error":"HTTPSConnectionPool(host='x', port=443): Read timed out. (read timeout=20)"} and {"meta":{"version":"v2.39.4","api_status":"stable"},"errors":[{"code":1014,"title":"Internal error","details":"Connection timed out. Please check if wacore is running: 10.0.14.50:6250"}]}.

    We are in contact with Meta's support to find the underlying causes and fix the issue. If you are affected by this issue, please create a Support Ticket with us (Category: Messaging Has Completely Stopped Working) and we will prioritize the restart for you.

Issue with Template Creation
  • Resolved
    Resolved

    The issue around template creation and submission has been resolved.

  • Identified
    Identified

    Upon the team's investigations we've identified the issue in an API rate limiting applied towards 360dialog from Meta. Our infrastructure has received high loads on the Partner API endpoint for creating message templates, which can not be propagated through Meta's systems, and is therefore getting blocked.

    We've applied some temporary changes in how we interact with Meta's infrastructure and continue to monitor the issue.

    New templates can still be submitted via the WhatsApp API, and will be synced to the 360 Client and Partner Hubs afterwards. View WhatsApp API endpoint documentation on docs.360dialog.com

  • Investigating
    Investigating

    We are aware of an issue that is impacting the Template Creation Process. Our engineering team is currently investigating it and we'll update again as soon as possible.

Problems with messaging
  • Resolved
    Resolved

    Meta have advised that the issue impacting message delivery is resolved.

    If you are still experiencing a problem relating to this incident: please create a Support Ticket with us (Category: Messaging Has Completely Stopped Working) and we will assist you.

    We are awaiting RCA from Meta.

  • Monitoring
    Monitoring

    Meta have advised that the issue impacting message delivery has now been resolved and the service has been restored.

    Some WhatsApp API setups require a restart to fully restore the service. We are currently working on restarting all setups that require this action.

    If you urgently require a restart, please create a Support Ticket with us (Category: Messaging Has Completely Stopped Working) and we will prioritize the restart for you.

  • Identified
    Identified

    Meta has confirmed that the problem is rooted in the WhatsApp Business Platform. We are working closely together with them to understand how to solve it as soon as possible. We will update this incident as soon as we have more information from Meta

    You can recognize this issue if you try to perform certain operations like check contacts or send messages and you receive a response such as: errors":[{"code":1014,"title":"Internal error","details":"Connection timed out. Please check if wacore is running: x"}]} or "error": "HTTPSConnectionPool(host='x', port=443): Read timed out. (read timeout=20)"

  • Investigating
    Investigating

    We are receiving some reports that there is an issue with the messaging service (such as {"error":"HTTPSConnectionPool(host='x', port=443): Read timed out. (read timeout=20)"} We are currently investigating.

360 Hub issue - export file missing data
  • Resolved
    Resolved

    The issue has now been resolved.

  • Investigating
    Investigating

    Some Partners are reporting missing data when exporting data files. We are aware of the issue and are currently investigating it.

    In the meantime, we recommend that you use our Partner API to retrieve the data.

Numbers disconnected from API (code 1014, Connection refused)
  • Resolved
    Resolved

    The issue has now been resolved

  • Investigating
    Investigating

    Due to a problem with Meta's security automation, some API deployments were disconnected from the WhatsApp server.

    If your number was affected, you will get an error such as: code: 1014, title: 'Internal error', details: 'Connection refused. Please check if wacore is running: when trying to perform operations.

    Alternatively, the health check will return:

    "gateway_status": "unregistered"

    If this happens, please create a Support Ticket with us (Category: Messaging Has Completely Stopped Working) and we will re-register the number as per Meta guidance.

Apr 2022

5xx Server Code Error
  • Resolved
    Resolved

    We are aware of the issue that is affecting messaging and some other services.

    Our engineering team is currently working on a fix for it

Issue with Meta Graph API
  • Resolved
    Resolved

    The issue is now resolved.

  • Identified
    Identified

    We are continuing to work with Meta Support to solve an issue with the Meta Graph API. Until a solution is found, the following services will be impacted:

    • Account syncing (status updates)
    • Template creation and approvals
    • Number Porting (migration to 360dialog)
    • Onboarding (Classic Signup flow)
  • Investigating
    Investigating

    We are aware of an issue with the Meta Graph API that is impacting the 360dialog Hub (number onboarding, migrations and status updates). Our engineering team is currently investigating with Meta Support.

Numbers disconnected from API (code 1014, Connection refused)
  • Resolved
    Resolved

    Meta advised that the issue should now be resolved.

  • Monitoring
    Monitoring

    We have identified and re-registered most affected numbers.

    If your number is disconnected, please create a Support Ticket with us and we will re-register the number as per Meta guidance.

  • Investigating
    Investigating

    Meta reported an error that might cause your API deployments to get disconnected from the WhatsApp server and unable to reconnect. https://status.fb.com/whatsapp-business-api

    If this happens, you will get an error such as: code: 1014, title: 'Internal error', details: 'Connection refused. Please check if wacore is running:

    If this happens, please create a Support Ticket with us and we will re-register the number as per Meta guidance.

Issue with Meta Graph API
  • Resolved
    Resolved

    The issue is now resolved.

  • Investigating
    Investigating

    We are aware of an issue with the Meta Graph API that is impacting the 360dialog Hub (number onboarding, migrations and status updates). Our engineering team is currently investigating with Meta Support.

Some disruptions to Embedded Signup ("number setting up > 10 mins")
  • Resolved
    Resolved

    The issue has now been resolved.

  • Investigating
    Investigating

    We are aware of an issue that is occasionally affecting the Embedded Signup process. If you get stuck on the 'number setting setup' screen for more than 10 mins, it is likely because the User Token required to complete the setup has expired. As a workaround, we recommend the following steps:

    1. Logout of Facebook

    2. Login to Facebook

    3. Redo the Embedded Signup Process

      Our engineering team is currently working on a way to improve this process.

    If the workaround fails, create a ticket with our Support Team and we will assist you.

Mar 2022

Message Delivery
  • Resolved
    Resolved

    Following Meta advise, we have rolled back all affected numbers to v2.37.2. We will continue to monitor the situation and automatically roll back any numbers that are affected by the issues described in the WhatsApp Business Platform Changelog

  • Identified
    Identified

    We are aware that some numbers are experiencing disconnects of the API client. This issue is limited to only certain orchestrations.
    We are following Facebook / Meta guidance and will downgrade these numbers to v2.37.2 instead.

  • Investigating
    Investigating

    Message from Meta: We are aware of an issue that may be impacting media message delivery. Our engineering teams are actively looking to resolve the issue as quickly as possible

Some disruptions to Embedded Signup
  • Resolved
    Resolved

    The Embedded Signup process has been stabilized.

  • Investigating
    Investigating

    We are aware of an issue that is impacting the Embedded Signup process. If the Embedded Signup flow is aborted before completion, we incorrectly receive a success callback from FB / Meta servers and the user will be logged into the 360dialog Hub. There, they will see a message "we are setting up your number" indefinitely.

    If this happens, we suggest:

    • Log out of the 360dialog Hub
    • Redo the ES flow

    Facebook / Meta are currently investigating the issue.

Message Delivery
  • Resolved
    Resolved

    Message from Meta: We are recovering from an earlier outage impacting message delivery, and services are in the process of being restored. We apologize for any inconvenience that this may have caused.

    Please update to our new version 2.39.3

    For more information please check our changelog: https://developers.facebook.com/docs/whatsapp/on-premises/changelog

  • Investigating
    Investigating

    Message from Meta: We are aware that business are unable to send messages to a consumer if the consumer initiated the WA thread between the two parties and the business sends a message after 7 days.

    The work-around is to remove the stale check from the send message path

Some disruptions to Ticket Service
  • Resolved
    Resolved

    Ticketing issue resolved.

  • Identified
    Identified

    You can still create tickets via the 360dialog Hub. You will receive support response notifications via email.

  • Investigating
    Investigating

    We are aware of an issue that is impacting our Ticketing Service in the 360dialog Hub. Some users cannot view ticket content. Our engineering team is currently investigating it and we'll update again as soon as possible.

Onboarding Issues - Embedded Signup
  • Resolved
    Resolved

    This issue is now resolved. If you experienced and issue with Embedded Signup - please re-submit your number.

  • Monitoring
    Monitoring

    We have identified the issue and deployed a fix. We are now monitoring the situation.

  • Investigating
    Investigating

    We are are receiving reports that the Embedded Signup process is stalling at the "setting up number" screen. We recommend that you PAUSE using Embedded Signup until further notice.

Mar 2022 to May 2022

Next