Design a Job Scheduler
Problem Statement#
"Design a distributed job scheduler. Users register work to run at a specific time or on a repeating schedule, the system runs it close to on time even at high volume, retries it when it fails, and lets them see what happened."
Cron on one box is twenty lines of config. Cron for ten thousand executions a second, where the box running it can die mid-job, is a different problem entirely.
Two words get used interchangeably here and keeping them apart is most of the design:
- A job is the definition. "Email the daily report to Priya, every weekday at 09:00."
- An execution is one instance of it. Monday's 09:00 run. Tuesday's 09:00 run. Each has its own status, attempt count and result.
Conflating them is the single most common mistake in this interview, and almost every hard part downstream gets easier once they are separate.
In this design you'll learn:
- Why storing a cron string and scanning for matches does not scale, and what to store instead
- How to run jobs within 2 seconds of their scheduled time without querying the database every 2 seconds
- What actually happens when a worker dies holding a job, and why "at-least-once" is the only honest promise
- Why a scheduler cannot give you exactly-once execution, and what it can give you instead
- Why 10,000 jobs a second is not evenly spread, and what midnight does to your fleet
Requirements Gathering#
The two answers that change the most here are how precise the timing has to be and what the system is allowed to promise about duplicate runs. Get those early.
Candidate: "What kinds of schedule do we support? One-off, future-dated, recurring?"
Interviewer: "All three. Run now, run at a timestamp, or run on a repeating cron schedule."
Candidate: "When a job runs, are we executing arbitrary user-supplied code, or calling a task type that was registered ahead of time?"
Interviewer: "Registered task types. The scheduler looks up a handler and invokes it with parameters. Sandboxing untrusted code is not your problem here."
Candidate: "How close to the scheduled time does a job have to start?"
Interviewer: "Within about 2 seconds."
Candidate: "Is that 2 seconds to start the job, or to finish it?"
Interviewer: "To start it. We don't control how long the task itself takes."
Candidate: "What scale? And is that jobs created or jobs executed?"
Interviewer: "Executions. Up to 10,000 a second. Creation is much lower, because a lot of jobs are recurring."
Candidate: "Do we need exactly-once execution, or is at-least-once acceptable?"
Interviewer: "At-least-once. Never silently skip a job. A duplicate run is survivable."
Candidate: "What happens when a job fails? Retries, and how many?"
Interviewer: "Retry with backoff, give up after three attempts, and make the failure visible."
Candidate: "Can users see history, or just the current state?"
Interviewer: "History. They should be able to list their executions and see status, timing and errors."
Candidate: "Are there dependencies between jobs, like Airflow DAGs where B runs after A succeeds?"
Interviewer: "Out of scope. Independent jobs only. Same for cancel and reschedule."
Functional and Non-Functional Requirements#
Functional Requirements#
- Users schedule a job to run immediately, at a future timestamp, or on a recurring cron schedule
- The system executes each scheduled occurrence close to its intended time
- Failed executions retry with exponential backoff, up to 3 attempts, then land in a terminal failed state
- Users list and inspect their executions: status, scheduled time, actual start, duration, error
Non-Functional Requirements#
| Requirement | Target |
|---|---|
| Timeliness | p99 starts within 2s of the scheduled time |
| Delivery | At-least-once. An execution is never silently dropped |
| Throughput | 10,000 executions/sec |
| Consistency | Availability over consistency. Duplicate runs beat missed runs |
| Durability | An accepted job survives the loss of any single component |
Out of scope: exactly-once execution, DAG dependencies between jobs, sandboxing untrusted code (see Design LeetCode for that), cancel and reschedule, per-tenant quotas.
Capacity Estimates#
That last block is the one worth saying out loud. The throughput number tells you nothing about the worker fleet without the job duration, which is why the SLO has to be about starting on time rather than finishing. The scheduler owns dispatch latency. The task owns its own runtime, and a slow task must not be allowed to delay an unrelated one.
High-Level Design#
Four things happen on different clocks: users create definitions, something turns definitions into dated occurrences, something hands occurrences to workers at the right moment, and workers run them.
Create. The API writes a job definition, and for a one-off it writes the single execution row too. It returns immediately; nothing waits on the job actually running.
Materialise. A background process expands each recurring definition into concrete execution rows a few hours ahead, so the system always has a dated list of what is coming rather than a pile of cron strings to evaluate.
Dispatch. Every few minutes the dispatcher asks the store for executions due in the next window and pushes them onto a queue that will not deliver them until their moment arrives.
Execute. Workers pull from the queue, run the handler, and write the outcome back. Failures get re-enqueued with a delay.
The idea holding it together: a durable store decides what should run, and a delay queue decides when. The store is too slow to consult every two seconds and the queue is too lossy to be the source of truth, so each does the half it is good at.
Deep Dives#
1. Definitions, Occurrences, and Why Cron Strings Don't Scale#
Start with the naive schema: one row per job, with the cron expression in a column. Now answer "what should run in the next minute?"
You cannot, without evaluating every cron expression in the table. There is no index that answers "does 0 9 * * 1-5 match 09:00 next Tuesday" because the answer is computed, not stored. At a few thousand jobs you can brute force it. At millions you cannot, and the cost grows with the number of jobs rather than with the number that are actually due.
The fix is to stop storing a rule and start storing dates. Expand each recurring definition into concrete execution rows covering a rolling horizon, say the next 6 hours. Now "what runs next" is a range scan on a timestamp, which every database is good at.
It is the same split a calendar app makes between a repeating event and the instances you see on each day, and the same one Design LeetCode makes between a problem and a submission.
Who creates the next occurrence matters more than it looks. The tempting answer is "when Monday's run finishes, create Tuesday's". Do not do this. It chains every future run to the success of the previous one: if Monday's execution row is lost, or the worker dies at exactly the wrong moment, or a bug marks it terminal, Tuesday never gets created and the job silently stops forever. Nobody notices until someone asks why the report stopped arriving.
A separate materialiser that keeps a rolling horizon has no such chain. It wakes up, asks "which definitions have no execution rows beyond time T", and fills the gap. If it misses a cycle it catches up on the next one. If it double-runs it writes nothing new, because (job_id, scheduled_for) is unique. Self-healing beats clever.
For senior interviews ↓ timezones, DST, and the runs that happen twice or never
"Every day at 02:30" is not a fixed interval, it is a wall-clock promise, and wall clocks do strange things twice a year.
In a timezone that springs forward, 02:30 does not exist on that date. In one that falls back, 02:30 happens twice. A materialiser that naively adds 24 hours will skip a run in spring and create a duplicate in autumn, and the duplicate is the dangerous one if the job moves money.
Store the schedule as a cron expression plus an explicit IANA timezone (Europe/London, not a UTC offset, because the offset itself changes). Compute each occurrence in that timezone using a real tz database, then persist the resolved instant as UTC in scheduled_for. Everything downstream works in UTC and never thinks about this again. For the ambiguous hour, pick a documented rule (fire once, on the first occurrence) rather than leaving it to whatever your date library happens to do.
This is also why the tz database version is a deployment concern. Governments change DST rules with a few months' notice, and a fleet running a stale tz database will schedule a whole country's jobs an hour off.
Also worth planning for: the backfill. If the materialiser is down for six hours, it wakes to find thousands of occurrences whose time has passed. Running them all at once is a stampede, and running none of them silently loses work. Give each job a policy: skip (fire only the latest, right for a "sync current state" job), or catch_up (run every missed occurrence, right for a job that processes a time window), with a cap on how far back it will go.
2. Hitting 2 Seconds Without Hammering the Database#
The store now holds dated rows, so the obvious dispatcher is a loop: every 2 seconds, ask for everything due in the next 2 seconds, and run it.
That fails for four compounding reasons. Your poll interval is a hard floor on precision, so a 2 second SLO needs a sub-2-second loop. Each of those queries returns ~20,000 rows at our volume. Reading and deserialising 20,000 rows takes hundreds of milliseconds before you have dispatched anything, eating the budget you were trying to hit. And you are now running the heaviest query in the system continuously, forever, against the store that everything else depends on.
Split it into two layers with different jobs. A slow, durable layer decides what is coming. A fast, in-memory layer decides when each one fires.
The database is now queried 12 times an hour instead of 1,800, and precision no longer depends on the poll interval at all. It depends on the queue, which is built for exactly this.
Which queue matters, because most of them cannot do it:
Wrong shape for this. A log delivers in append order within a partition, so a job scheduled for 10:00:05 that you write after one scheduled for 10:04:00 sits behind it. You cannot insert into the middle of a log, and "wait until this timestamp before delivering" is not something a log does.
Excellent for the resulting event stream (execution outcomes, audit, analytics). Not for the timing layer.
Score each entry by its execution timestamp. Workers poll ZRANGEBYSCORE key -inf now and atomically pop what is due. Sub-millisecond, trivially supports inserting a job that fires before things already queued, and you can inspect exactly what is pending.
You own the failure handling, which is the real cost: Redis is memory-first, so a node loss drops whatever was in flight, and you need your own claim-and-retry mechanism. Both are manageable (the store is still the source of truth, so a dispatcher re-poll refills the queue) but they are yours to build.
The right answer when managed services are off the table, which many interviewers will specify.
Native delayed delivery, and crucially the visibility timeout gives you worker-death handling for free (deep dive 3). Throughput is effectively unlimited for our volume, and there is no partitioning work to do.
The constraint that shapes the design: delays cap at 15 minutes. That is exactly why the dispatcher horizon is ~5 minutes rather than an hour. DelaySeconds is also a floor, not a guarantee, though with workers long-polling continuously the extra is negligible against a 2 second budget.
Decision: SQS if managed services are allowed, because visibility timeouts and dead-letter queues solve a whole deep dive for free. Redis sorted sets otherwise, accepting that you then build the claim and retry logic yourself. Either way the shape is identical, and being able to say "the horizon is 5 minutes because SQS caps delay at 15" shows you have actually used the thing.
The near-term job. A job created at 10:00:00 to run at 10:00:30 is invisible to a dispatcher that already polled at 09:58 and will not poll again until 10:03. It misses its time by minutes. Fix it in the API: after writing the execution row, check whether it falls inside the current horizon, and if so enqueue it immediately. The dispatcher covers everything beyond the horizon and the write path covers everything inside it.
For senior interviews ↓ two dispatchers, double enqueues, and whose clock is right
The write-through path and the dispatcher will both enqueue the same execution. Something created at 10:02:50 for 10:04:00 gets enqueued by the API, and then the 10:03 dispatcher poll sees it still marked pending and enqueues it again. Two deliveries, two runs.
Resist fixing this with coordination. Instead make it harmless: the worker's first action is a conditional claim, UPDATE executions SET status='running', attempt=attempt+1 WHERE execution_id=? AND status='pending', and a worker that claims zero rows drops the message. One atomic write settles the race with no locks and no leader election, and it is the same guard you need anyway for at-least-once redelivery.
Run more than one dispatcher, and let them overlap. A single dispatcher is a single point of failure, and "elect exactly one leader" is a much bigger commitment than this problem needs. Since duplicate dispatch is already harmless, run several, have each poll the full horizon, and let the claim sort it out. Availability without consensus, which is the trade the requirements asked for.
Whose clock decides "now"? Every component compares timestamps: the materialiser resolving occurrences, the dispatcher computing the delay, the worker claiming. A dispatcher whose clock is 10 seconds fast computes delays 10 seconds short and fires everything early, blowing the SLO in a way that looks like a queue problem. Keep NTP tight, alert on drift, and derive delays from the store's clock rather than each host's where you can.
3. At-Least-Once, and the Worker That Dies Holding Your Job#
Two very different failures hide behind "the job failed".
Visible failure. The handler throws. You know exactly what happened, so wrap execution in a try/catch, record the error, increment the attempt count, and re-enqueue with an exponential delay (5s, 25s, 125s). After three attempts mark it terminally failed and stop. Jitter the backoff, or a downstream outage that fails a thousand jobs at once will retry all thousand in the same second, three times.
Invisible failure. The worker is killed mid-execution. No exception, no error, no status update. From the outside it is indistinguishable from a job that is simply taking a while, and that ambiguity is the actual problem.
Ping workers, and when one stops answering, mark it dead. This tells you a machine is gone and nothing about which executions it was holding, so you still have to reconstruct that from somewhere, and you will race the worker that is unhealthy but still running. Detecting the symptom rather than reclaiming the work.
A worker claims an execution with a lease that expires, say 30 seconds out, and renews it every 10 seconds while it works. A sweeper reclaims anything whose lease has lapsed. Works on any store, gives you a clean answer for long jobs (keep renewing), and you can see live what is claimed by whom.
You build and operate it: the sweeper, the renewal loop, and the tuning. Worth it when you are not on a queue that does this for you.
The queue hides a message when it is delivered and redelivers it unless the worker explicitly deletes it. A dead worker never deletes, so the message reappears and another worker takes it. Exactly the semantics you want, already built, already battle-tested.
ApproximateReceiveCount gives you the attempt count and a dead-letter queue catches anything that exceeds the limit, so the whole retry apparatus is configuration rather than code.
Decision: visibility timeout when the queue offers one, with heartbeat-style extension for long jobs. Leases when it does not. They are the same idea, differing only in who operates it.
The tuning detail that bites everyone: the visibility timeout must exceed your longest legitimate job. Set it to 30 seconds, run a job that takes 5 minutes, and the queue redelivers at 30 seconds to a second worker while the first is still happily working. Now it runs twice, concurrently, and you did not even have a failure. Either set the timeout above the worst case, or have workers extend it as they go, which is the better answer because it adapts instead of guessing.
The sharp edge to name unprompted: you cannot distinguish "the worker died before doing the work" from "the worker did the work and died before acking". No amount of engineering removes that ambiguity, which is precisely why the promise is at-least-once and why deep dive 4 exists.
4. Idempotency, Because Duplicates Are Certain#
At-least-once means duplicates are not an edge case, they are a scheduled event. A refund job that runs twice refunds twice.
Works in testing, fails in production, and fails silently in a way that shows up in a customer's bank statement rather than in your logs.
Before running, conditionally insert execution_id into a dedup table, and skip if it already exists. Central, uniform, and needs no cooperation from task authors.
But the insert and the side effect are not atomic. Insert first and a crash between the two means the job never runs and never retries, turning at-least-once into at-most-once, which is worse. Insert after and a crash between the two means it runs twice anyway. The window shrinks, it never closes.
The scheduler passes a stable execution_id to the handler, and the handler uses it as an idempotency key against whatever it touches. A payment call sends it as the provider's idempotency key. A database write makes it a unique column. A file write names the output after it.
The duplicate still runs. It just cannot produce a second effect, because the effect is owned by the system that can actually make the check and the change atomic.
Decision: stable keys plus idempotent handlers, with the dedup table as a backstop for tasks that cannot be made idempotent. The framing worth stating: a scheduler can guarantee at-least-once delivery, and only the task can guarantee exactly-once effect. The scheduler's obligation is to hand every attempt the same stable key so the task can do its half. Promising "exactly-once execution" is promising something no distributed scheduler can deliver, and an interviewer who knows the space is listening for whether you know that.
The key must be the execution id, not the job id. Every attempt of Monday's run shares one key, and Tuesday's run gets a different one. Key on the job and Tuesday dedupes against Monday, and your daily report is delivered exactly once, ever.
5. Scaling the Write Path#
The executions table takes ~25,000 writes a second and is partitioned on time so the dispatcher can range-scan it. Those two facts fight each other.
If the partition key is an hourly bucket, every write for the current hour lands on one partition. DynamoDB caps a partition around 1,000 write units; Cassandra gets a hot node; Postgres gets one hot page and lock contention. You have built a system whose write throughput is one partition's worth, no matter how many nodes you own.
Shard the bucket. Append a suffix to the partition key so one logical hour becomes N physical partitions:
Writes now spread across 32 partitions, which comfortably covers 25k/sec. The cost lands on reads: the dispatcher must query all 32 shards for a bucket instead of one. That is fine, because it issues them in parallel, it runs 12 times an hour rather than continuously, and the shard count is fixed and known.
Pick the shard count deliberately. Too few and you have not solved the hot partition. Too many and every dispatch becomes a wide scatter-gather where the slowest shard sets your latency. 32 is a reasonable starting point at this volume, and it is a number you should be able to justify rather than a magic constant.
The status query needs its own index. "Show me my executions" cannot scan a table partitioned by time. Add a secondary index keyed by user_id and sorted by scheduled_for, which makes the user's view a single efficient range scan with free pagination. It costs extra write amplification on a table already doing 25k writes/sec, which is a real cost worth naming rather than waving through.
History is the storage problem. At ~650 GB/day, keeping everything hot forever is not a plan. Keep 90 days in the operational store for the UI and retries, and age older rows into object storage where they are cheap and still queryable for audit. Set this up on day one; retention is painful to add once the table is enormous.
For senior interviews ↓ noisy neighbours and poison jobs
One tenant can starve everyone. A customer who schedules 500,000 executions for 09:00 fills the queue ahead of everybody else's 09:00 jobs, and every other tenant misses the SLO because of one account. A single FIFO queue gives you no defence. Either shard the queue by tenant and consume round-robin, or cap in-flight executions per tenant at dispatch so one account cannot occupy the whole fleet. The unfairness is invisible until it happens and then it affects every customer at once.
Separate slow jobs from fast ones. A 30-minute job and a 200ms job in the same pool means the long ones occupy workers while short ones queue behind them. Route by expected duration into separate pools so a slow class cannot delay a fast one, and so each pool can autoscale on its own signal.
Poison jobs. A handler that reliably crashes its worker (an OOM, a segfault in a native dependency) takes the worker down with it, gets redelivered, and kills the next one. Left alone it walks through the fleet. The attempt cap is the defence, so it must be enforced by the queue and not by the handler, since the handler never gets to increment anything. After N delivery attempts the message goes to a dead-letter queue and a human looks at it.
Autoscale on queue depth and oldest-message age, not CPU, for the same reason as Design LeetCode: a busy worker is at 100% CPU by design, so CPU tells you nothing about whether you are keeping up. Backlog age does.
6. The Midnight Problem#
Everything above assumes 10,000 executions a second arriving smoothly. They will not.
Cron expressions cluster, hard, because humans write them. 0 * * * * puts every hourly job on the same second of the hour. 0 0 * * * puts every daily job on the same second of the day. Nobody schedules anything for 03:47. The load is not a flat 10k/sec, it is a low baseline with enormous spikes at the top of every minute, a bigger one at the top of every hour, and the largest of the day at midnight UTC, where daily, weekly and monthly jobs all land together.
Three things follow, and none of them are the usual answer.
Size for the spike, not the average. A fleet provisioned for 10k/sec falls hours behind at a midnight peak that is 50x that. Either you provision for the peak and waste money all day, or you accept a bounded delay at peaks and say so in the SLO, which is the honest engineering answer. Say which you are choosing.
Smear the schedule, where the user does not care. "Every hour" almost never means "at exactly :00". Offer it as the default: derive a stable per-job offset from hash(job_id) % 3600 and fire hourly jobs at that second instead of at zero. The spike flattens into a smooth line, the user's job still runs every hour, and nobody notices. Keep exact-time scheduling available for jobs that genuinely need it (a market open, a report with a contractual deadline) and treat it as the exception rather than the default. This one change does more for peak load than any amount of extra capacity.
Pre-scale on the schedule you already have. You are not guessing at demand. The executions table literally contains the next several hours of work, so you can count what is coming and scale the fleet ahead of it. A scheduler is one of the few systems that can see its own future load, and reactive autoscaling at midnight is scaling up after you are already behind.
Data Model#
- Unique on
(job_id, scheduled_for)so the materialiser is safely re-runnable and can never create a duplicate occurrence. - Secondary index:
user_idpartition,scheduled_forsort to serve the status view without touching the time-partitioned base table. - Executions older than 90 days age out to object storage, driven by the time bucket, which makes expiry a partition drop rather than a delete scan.
materialised_tois what makes the materialiser self-healing: it is a cursor, so a missed cycle is caught up rather than lost.
API Design#
Two things worth saying while you write this. Returning next_runs on create is not decoration: cron plus timezone is genuinely hard to get right, and showing the user the instants you resolved is how they catch a mistake before it fires at the wrong hour for three months. And user_id comes from the session, never the body, or anyone can schedule work as anyone else.
Failure Scenarios and Edge Cases#
For senior interviews ↓ what breaks and how the system recovers
The materialiser stops
Nothing fails immediately, which is what makes it dangerous. Executions already in the horizon keep running normally, and the system looks perfectly healthy right up until the horizon runs dry hours later, at which point recurring jobs stop with no error anywhere. Alert on the horizon itself: min(materialised_to) across enabled jobs should always sit comfortably in the future. That single metric is the one that catches this class of silent failure.
A dispatcher dies
Run several and let them overlap. Because the worker's conditional claim makes duplicate dispatch harmless, losing one dispatcher costs nothing and needs no leader election. The next poll from any surviving instance picks up everything still pending.
The queue is unavailable
Nothing is lost, because the queue was never the source of truth. Executions stay pending in the store and the dispatcher refills the queue once it recovers. Jobs run late, which the requirements allow, rather than not at all, which they do not. This is the payoff for keeping the durable layer and the timing layer separate.
A worker dies mid-execution
The visibility timeout expires, the message reappears, another worker claims it. The user-visible cost is the timeout's worth of delay. Since you cannot tell whether the work completed before the crash, the duplicate is handled by idempotency rather than prevented.
A job overruns its visibility timeout
The queue redelivers while the first worker is still running and you get two concurrent executions of the same occurrence, with no failure having occurred. Workers must extend the timeout on a heartbeat while they work, and the extension must be wired to real progress rather than a background timer that keeps renewing for a hung process.
A downstream dependency is down
Ten thousand jobs call an API that is failing, each retries three times with backoff, and your scheduler becomes a DDoS against a service that is already struggling. Put a circuit breaker per task type: after a threshold of consecutive failures, stop dispatching that type, park the executions, and resume on a probe. Jittered backoff alone is not enough when the failure is correlated across every job at once.
Clock skew
Timestamps are compared in several places, so a host whose clock is fast fires early and one that is slow fires late, both of which look like queue problems when you go looking. Keep NTP tight and alert on drift.
DST and timezone changes
"Every day at 02:30" can vanish or occur twice. Resolve occurrences in an explicit IANA timezone against a current tz database, persist the resolved UTC instant, and define the ambiguous-hour rule rather than inheriting whatever your date library does. Covered in deep dive 1.
A catch-up storm after an outage
The system comes back to thousands of overdue occurrences and tries to run them all at once. The per-job catch_up_policy decides whether each one is skipped or replayed, and the dispatcher rate-limits the replay so recovery does not become a second outage.
Code#
The dispatcher and the worker#
If you cannot use a managed queue#
A Redis sorted set is the delay queue, scored by execution time. The pop has to be atomic or two workers claim the same entry, so it runs as one script.
That gives you delayed delivery. It does not give you the visibility timeout, so a worker that dies after ZREM and before finishing drops the entry. The store is still the source of truth, so the next dispatcher poll re-enqueues anything left pending, but that delay is your real recovery time and you should size the poll interval knowing it.
Trade-off Summary#
| Decision | Chosen | Alternative | Why |
|---|---|---|---|
| Data model | Definitions and executions split | Cron string in one row | You cannot index a cron expression; expanding to dated rows makes "what's due" a range scan |
| Next occurrence | Materialiser with a rolling horizon | Create the next one when the current finishes | Chaining means one lost execution stops the job forever, silently |
| Timing | Slow durable poll plus a delay queue | Poll the database every 2s | Sub-2s polling returns 20k rows continuously and precision is still capped by the interval |
| Delay layer | SQS, or a Redis sorted set | Kafka | A log cannot insert ahead of queued entries or hold a message until a timestamp |
| Near-term jobs | Enqueue on the write path too | Shorten the poll interval | A job due in 30s cannot wait for the next poll; the horizon covers the rest |
| Dispatcher count | Several, overlapping | Leader election | Duplicate dispatch is already harmless once the claim is conditional; no consensus needed |
| Worker death | Visibility timeout with heartbeat extension | Health checks | Health checks find dead machines, not orphaned work |
| Delivery | At-least-once | Exactly-once | No scheduler can promise it; stable keys let the task deliver exactly-once effects |
| Idempotency | Stable execution_id handed to the handler | Dedup table in the scheduler | The dedup insert and the side effect can't be atomic; the task's own store can be |
| Partitioning | Time bucket plus 32-way shard suffix | Bare hourly bucket | Every write for the current hour on one partition caps throughput at one node |
| Peak load | Smear hourly jobs by a per-job offset | Provision for the midnight peak | Cron clusters at :00; spreading it is cheaper than 50x capacity that idles all day |
| Autoscaling | Queue depth and oldest-message age | CPU | A busy worker is pinned at 100% by design; backlog age is the real signal |
Follow-up Questions#
Basic
Q: Why not just store the cron expression and query for jobs that match?
Because a cron expression is a rule, not a value, so there is nothing to index. Answering "what runs in the next minute" means evaluating every expression in the table, and the cost scales with how many jobs exist rather than how many are actually due. Expanding each definition into dated execution rows turns the question into a range scan on a timestamp, which any database answers efficiently.
Q: Why two layers instead of just polling the database more often?
Your poll interval is a hard floor on precision, so a 2 second SLO needs a sub-2-second loop, and at 10k/sec each of those returns around 20,000 rows. Reading them takes hundreds of milliseconds before you have dispatched anything, and you are running your heaviest query continuously against the store everything else depends on. Polling every 5 minutes and letting a delay queue handle the final timing gives you both: the database is queried 12 times an hour, and precision comes from a component built for it.
Q: What breaks if you use Kafka as the delay queue?
Ordering. A log delivers in append order within a partition, so a job scheduled for 10:00:05 that you write after one scheduled for 10:04:00 waits behind it. There is no way to insert ahead of queued messages and no native "hold this until a timestamp". You want a priority queue keyed on time, which is a Redis sorted set or SQS with a delay. Kafka is a good fit for the resulting execution event stream, just not for the timing.
Q: How do you know a worker died rather than just being slow?
You don't, and that is the point. Externally the two are identical, which is why the mechanism reclaims work on a timeout rather than trying to diagnose the worker. The queue hides a message on delivery and redelivers it unless the worker explicitly deletes it, so a dead worker's job comes back automatically. The cost is that a slow-but-alive worker can have its job redelivered too, which is why workers extend the timeout on a heartbeat while they make progress.
Q: Why at-least-once and not exactly-once?
Because exactly-once is not available. A worker can complete the work and die before recording that it did, and nothing downstream can distinguish that from dying before starting. Given the choice between possibly running twice and possibly never running, a scheduler should pick running twice, then hand every attempt a stable execution id so the task can make its own effects idempotent.
Senior follow-ups ↓
Q: A recurring job silently stopped three weeks ago and nobody noticed. What happened?
Most likely the next occurrence was created on completion of the previous one, so the chain broke: one execution was lost or wrongly marked terminal, and there was never anything to trigger the next. This is why occurrences come from a materialiser keeping a rolling horizon rather than from each run creating its successor, since a horizon-based process catches itself up after any gap. The monitoring that catches it either way is an alert on min(materialised_to) across enabled jobs: if the horizon stops moving forward, something is wrong hours before any job is actually missed.
Q: "Every day at 02:30" in a timezone with daylight saving. What could go wrong?
Twice a year that wall-clock time is not a normal day. Springing forward, 02:30 does not exist and a naive add-24-hours skips the run. Falling back, 02:30 happens twice and you get a duplicate, which is the dangerous one if the job moves money. Store an explicit IANA timezone rather than a UTC offset (the offset itself changes), resolve each occurrence against a current tz database, persist the resolved UTC instant, and define a documented rule for the ambiguous hour instead of inheriting whatever your date library does. Keeping the tz database current is a deployment concern, since governments change the rules with a few months' notice.
Q: How do you stop one customer's 500,000 scheduled jobs from delaying everyone else's?
A single FIFO queue gives them the whole fleet the moment their jobs become visible, and every other tenant misses the SLO through no fault of their own. Cap in-flight executions per tenant at dispatch, or shard the queue by tenant and consume round-robin, so one account can only ever occupy its share. Worth pairing with separate pools for slow and fast job classes, so a 30-minute job cannot sit on a worker that a queue of 200ms jobs is waiting for.
Q: Your dispatcher and your write path can both enqueue the same execution. How do you handle it?
Don't prevent it, make it harmless. The worker's first action is a conditional claim, UPDATE executions SET status='running' WHERE execution_id=? AND status='pending', and whichever delivery loses writes zero rows and drops its message. One atomic write settles duplicate dispatch, queue redelivery and two workers racing the same message, with no coordination. It is also what lets you run several dispatchers without leader election, which removes a whole class of failure rather than managing it.
Q: The interviewer adds DAG dependencies: job B runs only after A succeeds. What changes?
Scheduling stops being purely time-based, so the trigger becomes an event. Only the roots of a DAG have a schedule; everything else has an upstream list and becomes eligible when all of its upstreams have succeeded for that logical run. You add a run id shared by every task in one pass of the DAG, a dependency edge table, and a step that re-evaluates downstream readiness when a task finishes. New failure modes come with it: a failed upstream blocks a subtree (so you need skip-versus-fail semantics and a way to resume from the failure point), and you have to reject cycles at definition time. The scheduler you have becomes the executor underneath a dependency resolver, which is essentially the line between cron and Airflow.
Minute-by-Minute Interview Playbook#
0 to 5 min: Requirements
Ask the questions above. The two that shape everything are the precision target (and whether it means start or finish) and at-least-once versus exactly-once. Pin down that the 10k/sec is executions, not creations, and confirm DAGs are out of scope so you do not accidentally sign up for Airflow.
5 to 10 min: Capacity estimates
Make three points. Definitions and executions differ by orders of magnitude, so they get sized separately. Execution history is the storage problem at ~650 GB/day. And 10k/sec does not size the worker fleet without a duration, which is why the SLO is on dispatch rather than completion.
10 to 18 min: Data model and high-level design
Lead with the definition-versus-execution split, because everything else depends on it, and say explicitly that you cannot index a cron expression. Draw create, materialise, dispatch, execute. Mention the materialiser as its own component and say why it is not "create the next one when this finishes".
18 to 40 min: Deep dives
The two-layer scheduler is the centrepiece: explain why sub-2-second polling fails, then split into a slow durable poll and a delay queue, and name why Kafka is the wrong shape. Then worker death and visibility timeouts, including the timeout-shorter-than-the-job trap. Then idempotency, and be precise that at-least-once delivery is the scheduler's job while exactly-once effect is the task's. Then the hot time partition and write sharding. If there is room, the midnight clustering point is the one most candidates never raise.
40 to 45 min: Failure and wrap-up
Cover the silent one (materialiser stops, horizon runs dry, alert on it) and the queue being down but nothing being lost because the store is authoritative. Close with: "A durable store decides what runs, a delay queue decides when, workers claim with a conditional update so duplicate delivery is harmless, and every attempt carries a stable id so tasks can make their effects idempotent."
Green flags
- Separates job definitions from executions before being prompted, and can say why cron strings aren't indexable
- Rejects "create the next occurrence on completion" because of the silent-stop failure
- Derives the two-layer scheduler from the cost of sub-2-second polling rather than reciting it
- Knows a log-based queue is the wrong shape for delayed delivery
- Says at-least-once is the only honest promise, and that idempotency belongs to the task
- Notices the visibility timeout must exceed the longest job
- Spots that the time-bucket partition key is a hot partition and shards it
- Raises cron clustering at midnight without being asked
Red flags
- Stores cron expressions and scans the table for matches
- Polls the database every second or two and calls the SLO met
- Claims exactly-once execution
- Has no story for a worker dying mid-job
- Uses a bare hourly bucket as the partition key at 25k writes/sec
- Retries with fixed backoff and no jitter after a correlated downstream failure
- Treats 10k/sec as flat load and sizes the fleet on the average
Further Reading#
Try a different category