Incident Postmortem Slow Query Development Server Track

Incident summary

Over the past week, when testing slow queries on the development track server, the queries have always been slower than production.

A bug in this slow query caused the development server down.

Fault

I've tried query optimization, and when run on the production server it runs perfectly.

The resource CPU development server beetween 06.00 - 18.00 always 100%

Impact

This incident had an impact on the testing stage, web tracking access was disrupted.

Detection

This incident was detected when, I was doing QEP on the development database. And I checked the index in the orders table, it turns out it is not the same as the production table.

Response

I myself (abdan) immediately handled the issue

Recovery

I used a three-pronged approach to the recovery of the system:

• Optimize the query
• Equate the index on the development server with the production server

The result, resource CPU server is normal on 05-12-2023

Timeline

All times are WIB.

November - The development server frequently goes down

Desember - Fixing slow query and add index on development server