360dialog - Reports of messages & notifications not arriving for Cloud API numbers – Incident details

All systems operational

Reports of messages & notifications not arriving for Cloud API numbers

Resolved
Degraded performance
Started over 2 years agoLasted about 3 hours

Affected

Meta Infrastructure

Degraded performance from 5:10 PM to 7:25 PM, Operational from 7:25 PM to 8:25 PM

WhatsApp Cloud API (Message Delivery)

Degraded performance from 5:10 PM to 7:25 PM, Operational from 7:25 PM to 8:25 PM

Updates
  • Resolved
    UTC
    Resolved

    The issue is now resolved.

    ---

    Post Mortem:

    Issue description and impact:

    On 14.03.2024 at ~16:30 we received reports from some Cloud API customers that messages and status webhooks were not arriving.

    Cause of the issue:

    We identified the cause of the issue as a recent deployment of a new Meta feature (phone number webhook) that was not working as described.

    Mitigation Steps / Summary:

    We rolled back the deployment reverting all numbers to a functioning state. This rollback was completed by ~18:40

    Prevention of recurrence & suggestions:

    We have completed an internal review with the relevant teams and

    made changes / improvements to how we test and deploy new features from Meta.

  • Monitoring
    UTC
    Monitoring
    A mitigation to restore functionality has been deployed. We are continuing to monitor the situation.
  • Identified
    UTC
    Identified
    Our engineering team have identified the issue and work is in progress to mitigate it. Note: the issue only affects a small % of Cloud API numbers that are currently using '/waba_webhook' (webhook per WABA) to receive inbound notifications.
  • Investigating
    UTC
    Investigating
    We are currently investigating reports about messages & notification delivery issues on Cloud API. Our engineering team is currently investigating it. We do not currently have a resolution ETA but we'll update again as soon as possible.