Cpu Utilization spiked to near saturation potentially leading to incident with higher db response times

Incident Report for Tonal

Resolved

Our care admin tool has a leak where in if someone puts in a 0 it leads to a massive table scan on the backend database.

Immediate resolution was switching read endpoints from write to read db.

Long term: backend team would consider adding an Elastic Search instance to our stack to support search through our user_profile, devices, subscriber and subscription models.

Posted May 25, 2022 - 16:39 PDT