Avoid 2:00 and 3:00 am cron jobs (2013)

Original Article ↗ Hacker News Discussion ↗

UTC vs local time on servers

  • Strong support for setting server clocks to UTC and converting to local time only at display or application level.
  • Counterpoint: for teams and systems that are strictly local (e.g., all-Pacific or single-HQ enterprises), local server time can simplify reading raw logs and business reasoning.
  • Several war stories where non‑UTC server time and per‑customer timezones caused recurring DST chaos; some organizations never fully fixed it.

Business requirements: “local midnight”, quotas, and reports

  • Many jobs are tied to human expectations: daily quotas at “midnight”, reports at “8am local”, market/regulatory cutoffs, shift-based operations.
  • UTC scheduling doesn’t remove the need to decide what happens on days with 23 or 25 hours or duplicated times (e.g., 01:30 occurs twice).
  • Suggestions include:
    • Rolling windows for quotas (e.g., last 24h) instead of calendar days.
    • Treating “run once per local day” as a business rule that must explicitly define behavior on DST transition days.

Cron, DST, and scheduler behavior

  • Core article’s bug was in vixie‑cron; some distros (e.g., Debian) added logic long ago to avoid double/zero runs across small clock changes.
  • Other cron variants (busybox, systemd timers) behave differently; some handle DST better, some are naive.
  • Several argue for avoiding cron entirely and using application‑level schedulers that are timezone/DST‑aware and user‑configurable.

Operational scheduling practices

  • Common advice:
    • Avoid 02:00–03:00 in DST zones; also avoid 00:00 and on‑the‑hour times to reduce ambiguity and contention.
    • Use odd minutes and randomized delays (or systemd’s RandomizedDelaySec / FixedRandomDelay) to spread load.
    • For “once per day” logic, some propose hourly cron + “has the day changed?” checks, though others see this as re‑implementing a scheduler.
  • anacron is recommended for “sometimes on” machines; it locks jobs and avoids double runs.

Logs, tooling, and human factors

  • Many prefer logs in UTC for cross‑team correlation; others like local timestamps for fast mental parsing.
  • Compromise patterns:
    • Store UTC (possibly with Unix timestamps) and attach or derive a local representation at view time.
    • Use tools/wrappers (e.g., tztail‑style) to convert while tailing.
  • Frustration with UIs that mix timezones, infer format from browser locale, or make UTC/24h usage harder than necessary.

Concurrency, robustness, and idempotency

  • Multiple comments stress that overlapping jobs are a more general problem than DST:
    • Use lockfiles/flock, semaphores, or idempotent job design so double runs are safe.
    • Treat downtime and restarts as cases where “once per day” semantics must still hold.

DST, timezones, and broader debates

  • Widespread dislike of DST; recurring arguments over permanent standard vs permanent summer time, or abolishing timezones vs keeping them.
  • Some point out that any “simple” replacement has already been tried somewhere and produced its own issues.
  • Technical purists mention TAI and leap seconds, but others note the interoperability cost when the rest of the world uses UTC.