Avoid 2:00 and 3:00 am cron jobs (2013)

Original Article ↗ Hacker News Discussion ↗

Running scheduled jobs around 2–3 a.m. can misfire when daylight saving time shifts cause hours to be skipped or repeated, leading to backups, reports, or billing tasks running twice or not at all. Commenters debate whether servers should always use UTC, local “business time,” or more sophisticated schedulers, and highlight trade‑offs between technical simplicity, user expectations, and global operations. Many advocate storing timestamps in UTC and handling localization at the edges, while also noting that DST itself introduces systemic complexity far beyond cron.

UTC vs local time on servers

  • Strong support for setting server clocks to UTC and converting to local time only at display or application level.
  • Counterpoint: for teams and systems that are strictly local (e.g., all-Pacific or single-HQ enterprises), local server time can simplify reading raw logs and business reasoning.
  • Several war stories where non‑UTC server time and per‑customer timezones caused recurring DST chaos; some organizations never fully fixed it.

Business requirements: “local midnight”, quotas, and reports

  • Many jobs are tied to human expectations: daily quotas at “midnight”, reports at “8am local”, market/regulatory cutoffs, shift-based operations.
  • UTC scheduling doesn’t remove the need to decide what happens on days with 23 or 25 hours or duplicated times (e.g., 01:30 occurs twice).
  • Suggestions include:
    • Rolling windows for quotas (e.g., last 24h) instead of calendar days.
    • Treating “run once per local day” as a business rule that must explicitly define behavior on DST transition days.

Cron, DST, and scheduler behavior

  • Core article’s bug was in vixie‑cron; some distros (e.g., Debian) added logic long ago to avoid double/zero runs across small clock changes.
  • Other cron variants (busybox, systemd timers) behave differently; some handle DST better, some are naive.
  • Several argue for avoiding cron entirely and using application‑level schedulers that are timezone/DST‑aware and user‑configurable.

Operational scheduling practices

  • Common advice:
    • Avoid 02:00–03:00 in DST zones; also avoid 00:00 and on‑the‑hour times to reduce ambiguity and contention.
    • Use odd minutes and randomized delays (or systemd’s RandomizedDelaySec / FixedRandomDelay) to spread load.
    • For “once per day” logic, some propose hourly cron + “has the day changed?” checks, though others see this as re‑implementing a scheduler.
  • anacron is recommended for “sometimes on” machines; it locks jobs and avoids double runs.

Logs, tooling, and human factors

  • Many prefer logs in UTC for cross‑team correlation; others like local timestamps for fast mental parsing.
  • Compromise patterns:
    • Store UTC (possibly with Unix timestamps) and attach or derive a local representation at view time.
    • Use tools/wrappers (e.g., tztail‑style) to convert while tailing.
  • Frustration with UIs that mix timezones, infer format from browser locale, or make UTC/24h usage harder than necessary.

Concurrency, robustness, and idempotency

  • Multiple comments stress that overlapping jobs are a more general problem than DST:
    • Use lockfiles/flock, semaphores, or idempotent job design so double runs are safe.
    • Treat downtime and restarts as cases where “once per day” semantics must still hold.

DST, timezones, and broader debates

  • Widespread dislike of DST; recurring arguments over permanent standard vs permanent summer time, or abolishing timezones vs keeping them.
  • Some point out that any “simple” replacement has already been tried somewhere and produced its own issues.
  • Technical purists mention TAI and leap seconds, but others note the interoperability cost when the rest of the world uses UTC.