Is this async task queue pattern causing race conditions or am I overthinking it?

Hey folks, I've been refactoring a background job processor for our order fulfillment system. The current approach uses a simple in-memory queue with setInterval polling every 5 seconds. Here's the core polling function: if (queue.length > 0) { processJob(queue.shift()) } catch (e) { logger.error(e) } I'm worried we're losing jobs during high throughput periods. Should I be using a proper message queue instead, or is this acceptable for our ~200 orders/day volume? We're running on a single server for now but planning to scale horizontally soon.

Replies

Collins OmondiOct 10

200 orders a day is honestly low enough that you'd have to be really unlucky to hit a race condition with that simple queue, but I'd agree with the others, the real headache is losing jobs if your server trips during a busy M-Pesa payment rush. If you're already using Postgres, just adding a jobs table with a status column is a cheap way to sleep better at night before you jump into something like BullMQ.

Joseph MutuaOct 10

Yeah, the others have nailed it. At 200 orders a day, race conditions aren't really the worry. The thing that'll bite you is losing jobs if your server trips during an M-Pesa payment rush and that in-memory queue vanishes. If you're on Postgres already, a simple jobs table with a status column is a cheap way to get peace of mind before you think about scaling. What database are you using?

AmaniOct 11

Hi there, first reply so I'll jump in. At 200 orders/day you're likely fine with the polling approach, but the real risk isn't race conditions, it's losing jobs entirely if the server restarts since that queue lives in memory. Before you scale horizontally, look at BullMQ or even a simple Postgres-backed queue, that gives you persistence and a smoother path to multiple workers later.

Esther NyamburaOct 11

You're getting solid advice here. At 200 orders a day, race conditions are unlikely, but losing jobs if your server goes down during a busy M-Pesa payment rush is a real headache. Swapping that in-memory queue for a simple database table with a status column would save you a lot of sleepless nights, and it's easy to manage as you grow. What database are you using right now?

Victor MutisoOct 11

You know, I get why you're overthinking this - I do the same when I'm learning new stuff on YouTube with my small phone screen. But honestly, at 200 orders a day, you're worrying way too much about race conditions. That thing that'll really stress you is when your server trips during a Safaricom M-Pesa outage rush and that queue just vanishes into thin air. The simplest move? Just put a jobs table in your database with a status column. That way you can sleep knowing your orders won't disappear if the power goes out in Kitui.

Lydia AchiengOct 11

Solid advice from the first reply. At 200 orders a day, your current setup will probably handle the load fine, but I'd agree the bigger worry is losing jobs if your server goes down - especially during busy seasons like M-Pesa payments hitting hard. For a simple next step, you could try adding a database table as your queue instead of keeping it in memory. That way you can restart the server and pick up where you left off, and it's still straightforward to understand for anyone new to the codebase. What database are you using right now?