360Dialog - Disruptions to messsaging – Incident details

All systems operational

Disruptions to messsaging

Resolved
Degraded performance
Started about 3 years agoLasted 13 days

Affected

360dialog Infrastructure

Degraded performance from 9:23 PM to 9:40 AM

360dialog WhatsApp Business API

Degraded performance from 9:23 PM to 9:40 AM

Meta Infrastructure

Degraded performance from 9:23 PM to 9:40 AM

WhatsApp On-Premise API (Message Delivery)

Degraded performance from 9:23 PM to 9:40 AM

Updates
  • Resolved
    Resolved

    Following Meta advice, we have mitigated the issue arising from media sending by rolling back affected numbers to v2.37.1.

  • Update
    Update

    Meta have confirmed there is a bug related to media message sending with WhatsApp Business API v2.39.1. Following Meta advice, we are preparing to roll back to v2.37.1.

  • Monitoring
    Monitoring

    Issues with Disconnecting Core-Apps A small number of WA Business API Client Core-apps are being disconnected due to an issue with Media handling. We are working with META Support and hope to have a solution soon.

  • Identified
    Identified

    Issues with Disconnecting Core-Apps We have identified an issue which is resulting some Core-apps being disconnected. We are working with META Support to find a solution.

  • Update
    Update

    Issues with Contacts and Messaging Some Businesses reported that message sending was failing (usually with a 1006 Unknown Contact error) even though the recipient contact was valid and a session (24 hour customer care window) was open.

    This issue appears to be related to the recent WABA application upgrade to v2.39.1 We discovered that the contacts database was re-created during the upgrade resulting in a new database table for the ContactStore.

    This is something we were not aware of as previous upgrades did not behave in this manner. META did not mentioned it in the upgrade notes or change logs. As a workaround, we recommend checking contacts (/v1/contacts) before messaging.

    Issues with Disconnecting Core-Apps Some Businesses reported issues that relate to disconnecting WA Coreapps. A disconnected Coreapp can be identified by calling the v1/healthcheck endpoint which will return a disconnected status. Symptoms include an error response such as "Connection timed out. Please check if wacore is running" The reason for some Coreapps being disconnected is still not 100% clear at this stage. META Support have implementing enhancing logging on reported numbers to help identify the root cause.

    However, it does appear to relate to Media handling - specifically how media URLs are uploaded to the Media volume. As a workaround, we suggest that you use media-ids to send media as opposed to media URLs.

  • Monitoring
    Monitoring

    We are still investigating the issue. We will update again as soon as we have more information.

  • Identified
    Identified

    We are receiving reports that some numbers are experiencing connectivity issues with the WhatsApp API. Our engineering team is currently troubleshooting with Facebook Support and we'll update again as soon as possible.