We had a severe load issue on our Bots backend services preventing Bots from being functional on your websites.Conversations handled exclusively by humans were still functional. However, if a bot intervened in the engagement flow used by visitors, the conversation could not take place.
This load issue was caused by an unscheduled self-cleaning script performed on the Bots database engine. This cleaning was performed on a table with a large amount of data. As a consequence, critical queries needed by Bots were not able to perform in reasonable time, making the whole Bots system degraded.
This issue happened on October 26th between 11:10 to 15:47 CEST.
Once started, the self-cleaning script cannot be stopped and must be completed. So we looked for alternative solutions.
In order to mitigate and restore the Bots services, we performed following actions: