Duplicate cron execution across scaled instances
Scaling a service from one replica to five quietly turned one scheduled job into five. Here's the fix.
The symptom
Duplicate email notifications, and duplicate payment execution, at the top of every hour. Nothing in the application code looked wrong — the job ran, did its work, and finished cleanly. It just ran more than once.
Root cause
The cron trigger lived inside the application container itself. That was fine at one replica. It stopped being fine the moment the service scaled to five — every replica has its own copy of the container, so every replica independently fired the same scheduled job at the same time.
Nothing coordinated the instances. Each one had no way of knowing the other four existed.
The fix
Cron execution moved out of the application container and into a single isolated worker, so the job is only triggered once regardless of how many application replicas are running.
As a second layer of protection, job execution is wrapped in a Redis-backed distributed lock using atomic TTL acquisition — SET key value NX PX — so even if the trigger source is ever duplicated again, only one instance can actually acquire the lock and run the job.
Takeaway
"Runs once" is not a property you get for free once you're running more than one instance — it's something you have to build in explicitly, either by owning the trigger or by coordinating through a lock.