Engineering Notes

Notes on the systems I build

Short write-ups on production problems I've actually run into — root causes, fixes, and what I took away from them.

Duplicate cron execution across scaled instances

Scaling a service from one replica to five quietly turned one scheduled job into five. Here's the fix.

Peak-traffic 500s that looked like a pooling problem, but were really a query problem wearing a pooling costume.

Customers were debited and still showing as unpaid, because the webhook handler was racing against the provider's timeout.