How We Run Cron Jobs - Every Second, Not Every Minute
Introduction
Cron jobs are automated tasks that run on a schedule — traditionally once per minute using Linux’s built-in cron system. That’s perfect for things like sending out a daily report or checking for updates. But at Teruza, we build real-time financial systems where a one-minute delay could mean missed trades, outdated data, or lost revenue.
To solve this, we built a custom scheduling engine that doesn’t just run “every second” — it runs continuously, restarting the moment a task finishes. Each task queue operates like its own high-speed conveyor belt, ensuring lead syncing, quote generation, webhook responses, and real-time monitoring without delay.
--- Ardi Coetzee, Chief Technology Officer at Teruza"We couldn’t afford to be a minute late. That’s why we built a scheduler that thinks — and runs — in seconds. It’s now core to how Teruza powers fast-moving businesses."
The Problem
Financial systems — especially trading platforms — can’t rely on data that’s even a few seconds out of date. Quote engines need live market information. Trade opportunities disappear in an instant. And compliance-related checks often depend on accurate, immediate event detection. Running cron jobs once a minute introduces unacceptable lag across the board.
We also found that in systems like ours, it’s not enough to simply “run more often.” A truly modern approach to task scheduling must run indefinitely — triggering jobs again as soon as the last one completes, without waiting for an arbitrary schedule. Moreover, it must support multiple independent queues, allowing different types of jobs (fast, slow, high-priority, tenant-specific) to be executed in parallel — without blocking each other.
Our Solution
We designed a custom cron engine that breaks free from traditional limitations. At its core is a process-based queue system, where each queue is tied to a dedicated Linux process. These processes don’t sleep between runs — they stay alive and restart the next job as soon as the previous one completes. This ensures truly continuous execution.
Jobs are grouped by type (e.g., webhooks, sync, billing, reporting) and assigned to separate queues. That way, a slow job in one queue won’t block a faster job in another. Need more throughput? Just spin up additional queues — the system is horizontally scalable.
To solve early challenges with high CPU usage and process churn, we implemented a long-polling model. Instead of spawning new processes constantly, queues now run persistently and check for new work at short intervals. This reduced load and dramatically improved system stability and essentially made it possible to have as many queues as you could possibly need.
For visibility, we also built a scheduling dashboard that lets admins view queue health, job timings, and trigger frequency in real time.
Impact
- Teruza systems now execute background tasks with virtually no delay, tailored to each client’s needs.
- Each Teruza instance can run multiple isolated queues, ensuring flexible and uninterrupted performance.
- Our long-running cron processes use fewer system resources while maintaining high throughput.
- Admins can monitor and tune cron behavior through a real-time UI — including queue priorities, error handling, and retry logic.
- Developers have the power to define how often jobs run — every 5 seconds, 30 seconds, or immediately after the previous task finishes — giving them total control over system responsiveness.
Reach out by submitting
the form below
Contact Information
Whether you are looking for a full-scale system or quick advice, we are always happy to help.
Reach out. We will make it worth your time.
Phone
Let's Talk
Ready to chat? Click the button below to book a time that suits you.
Book A Call Book A Call
Our Latest Projects
Here's What We've Been Up To Lately
work process
See the value in
our Development
Process
01Step
Project Discovery
Our Project Managers collaborate with your team to fully understand your unique needs.
02Step
Strategy Development
We formulate a proposal outlining exactly how we see the project panning out as a whole or in phases.
03Step
Implementing the solution
Our very transparent process ensures clients are kept informed regarding timeframes and budget throughout the project.
04Step
Continuous Monitoring
We work with clients to continuously monitor progress as well as changes to all environments.
Book a call with one of our Project Managers today to see how
Teruza can assist you with your development needs and ultimately boost
your projects potential.
Teruza can assist you with your development needs and ultimately boost
your projects potential.
