Community
Kevin Mwangi
Kevin MwangiTech
Oct 8

I learned the hard way why indexes matter in production

Last year I had a M-Pesa reconciliation endpoint that started timing out once we passed 50k transactions. The query was a simple date range scan on an unindexed created_at column. The fix was one CREATE INDEX statement. That single missing index took the query from 3 seconds to 12ms. The scary part is I knew better, I just got lazy in the rush to ship. Anyone else got a story about a small oversight that caused a big outage? Or a database indexing mistake that taught you something? Would love to hear what keeps you up at night in production.

7

Replies

7
Sarah Njeri
Sarah NjeriOct 8

Oof, that exact created_at index got me too, but on a client's WooCommerce site. Their monthly revenue report took forever to load and we blamed the plugin. Turned out the post_date column in wp_posts had no index. One index later and loads went from 45 seconds to under a second. Now I always ask devs to check the explain plan before they tell me 'it's just slow'. Gets me every time.

Halima Abdi
Halima AbdiOct 9

Sarah's right about checking explain plans early. Sometimes the simplest thing hides in plain sight. My lesson was on a community database for our skills program. We had a huge table of applicants and kept adding filters for reports, but nobody indexed the county column we filtered on most. Everything was fine in dev with a few hundred rows, then it ground to a halt during intake season when we hit 8k submissions. Same story, one index and it was night and day. Now I always ask our dev partner 'what's the biggest table and what are we searching on most?' before we build anything new.

Achieng Odhiambo
Achieng OdhiamboOct 9

Sarah's WooCommerce story and Halima's point about asking what you search on most really hit home. At the NGO we had a dashboard tracking malaria stockouts that would freeze every Monday morning. The report was scanning the whole stock_movements table (60k+ rows) with no index on facility_code, the column we filtered on every single time. One index took it from 8 seconds to 200ms. Now before I build any dashboard I sit with the team and ask 'which column will you filter on 90% of the time?'. It forces us to think about indexing early instead of as an afterthought.

Dennis Kipkorir
Dennis KipkorirOct 9

Great stories, everyone. In fintech what keeps me up at night is the compound index on (status, created_at). Get the column order wrong and the planner quietly ignores it, and everything looks fine until traffic spikes.

Faith Chebet
Faith ChebetOct 11

Your story hit close to home. At my boutique I had a customer database that would freeze when I tried to pull reports by location. Turns out I had no index on the town column - and we filter by town all the time for delivery planning. One index later and the query dropped from 15 seconds to under half a second. Now I always think about what columns we actually search on before adding any new table.

Lydia Achieng
Lydia AchiengOct 11

Your stories have me double-checking my own indexes tonight. For beginners reading this - the simplest takeaway is just ask yourself "which column will I search by most?" before you build anything. That habit alone would have saved me a few late nights. Anyone else got a "I knew better but shipped anyway" moment?

AM
AmaniOct 13

These stories are gold, and the pattern is always the same: we don't think about indexes until the data grows. One concrete habit that's saved me since my own outage is setting a weekly alarm to review slow query logs, even when nothing seems wrong, that's how you catch the next one before your users do.