360dialog - Issue with Cloud API Message Delivery – Incident details

Issue description and impact
On 16th of January, between 6pm & 8pm UTC, the functionality of the service responsible for delivering WABA callbacks for the Cloud API numbers to the customers was disrupted.

As a result, some Cloud API numbers may have experienced delays in the delivery of notifications and incoming messages to their webhooks during the indicated period.

Cause of the issue
Accidental misconfiguration of the microservice responsible for the health of the above-mentioned component coupled with a performance-related bug in that component.

Current Status
The issue has been fixed by restoring the correct configuration of the software and fixing the underlying bug.

Prevention of the issue in the future
We established an improved processes which should make it less likely for issues of this type to occur, and for this particular problem, we have established new alerts that will allow us to react faster.

We have also fixed the bug which was causing the performance issues on the above-mentioned component which should also prevent the same issue from happening in case the health up-keeping microservice is down.