All systems operational

Resolved
Issue with Cloud API Message Delivery

Started
January 16, 2024 at 8:00 PM
Status
Resolved after 6 days

Impact

Operational
Affected
Meta Infrastructure
WhatsApp Cloud API (Message Delivery)
  • Resolved
    Resolved

    Issue description and impact
    On 16th of January, between 6pm & 8pm UTC, the functionality of the service responsible for delivering WABA callbacks for the Cloud API numbers to the customers was disrupted.

    As a result, some Cloud API numbers may have experienced delays in the delivery of notifications and incoming messages to their webhooks during the indicated period.

    Cause of the issue
    Accidental misconfiguration of the microservice responsible for the health of the above-mentioned component coupled with a performance-related bug in that component.

    Current Status
    The issue has been fixed by restoring the correct configuration of the software and fixing the underlying bug.

    Prevention of the issue in the future
    We established an improved processes which should make it less likely for issues of this type to occur, and for this particular problem, we have established new alerts that will allow us to react faster.

    We have also fixed the bug which was causing the performance issues on the above-mentioned component which should also prevent the same issue from happening in case the health up-keeping microservice is down.

  • Monitoring
    Monitoring

    We are recovering from an issue that affected Cloud API message delivery from ~5:05pm to 6:21pm UTC. The situation has stabilized and we are continuing to monitor.