Maestro: Netflix's Workflow Orchestrator

Original Article ↗ Hacker News Discussion ↗

Relationship to Conductor and Netflix Ecosystem

  • Maestro is described as a domain‑specific implementation for ML and data workflows built on top of Conductor.
  • Conductor’s original Netflix repo is archived; active development moved to a community fork, but Netflix states Conductor is still heavily used internally and has grown in usage.
  • Metaflow is said to sit on top of Maestro, using it as the orchestration backend for Python-defined workflows.

Positioning vs Other Orchestrators (Airflow, Temporal, etc.)

  • Multiple commenters ask whether Maestro is an Airflow replacement; others see it as more comparable to Argo Workflows/Events or other DAG tools.
  • Some argue it’s more an alternative to Airflow than to Temporal, since Temporal is “workflow as code” while Maestro uses a JSON/DSL model.
  • Others note overlap with projects like Dagster, Prefect, Argo, Nextflow, Snakemake, Kestra, Windmill, etc., and question if Maestro adds much beyond “yet another orchestrator.”

Design, Features, and Stack Choices

  • Maestro supports both acyclic and cyclic workflows, with patterns like foreach, subworkflows, and conditionals; this is contrasted with “traditional DAG-only” tools.
  • Business logic can be packaged in containers, scripts, SQL, notebooks, etc.
  • Written in Java, using Conductor core and CockroachDB for state; this draws comparisons to Rust/Go-based alternatives and PostgreSQL-based designs.
  • Some praise the project as “complete” and promising; others criticize sparse documentation and marketing-heavy language in the blog post.

Trust, Longevity, and OSS Strategy

  • Several commenters warn against relying on Netflix OSS, citing a history of archiving or de‑prioritizing projects and keeping more advanced internal forks.
  • Others counter that many major OSS tools began as internal projects (e.g., Airflow), and that open-sourcing even imperfect tools still benefits the ecosystem.

Why So Many Workflow Engines?

  • Long subthread debates why companies keep building new orchestrators:
    • Problem space is broad (scheduling, dependencies, retries, domain awareness, observability).
    • Existing tools are seen as hard to operate, inflexible, or mismatched to local needs.
    • Some view generic workflow engines as a “design smell,” others see them as critical infrastructure still lacking a clear “Kubernetes/Terraform-style” winner.