What Dagu Does ​

Dagu is a command-native workflow engine that runs as a single binary with no external databases or message brokers. It turns scripts, commands, containers, server tasks, and agent CLIs into DAGs (Directed Acyclic Graphs) defined in YAML. It supports local execution, cron scheduling, queue-based concurrency control, and distributed coordinator/worker execution across multiple machines over gRPC.

All state is stored in local files by default. There is nothing to install besides the binary itself.

Real-World Use Cases ​

Dagu is useful when scripts, containers, server jobs, data tasks, or agent-driven work need visible dependencies, schedules, logs, retries, and a simple way to operate them.

Architecture ​

Dagu runs in three configurations:

Standalone. A single dagu start-all process runs the HTTP server, scheduler, and executor. Suitable for single-machine deployments.

Coordinator/Worker. The scheduler enqueues jobs to a file-based queue, then dispatches them to a coordinator over gRPC. Workers long-poll the coordinator for tasks, execute DAGs locally, and report status back. Workers can run on separate machines and are routed tasks based on labels. Mutual TLS secures gRPC communication between coordinator and workers.

Headless. Run without the web UI (DAGU_HEADLESS=true). Useful for CI/CD environments or when Dagu is managed through the CLI or API only.

Standalone:

  ┌─────────────────────────────────────────┐
  │  dagu start-all                         │
  │  ┌───────────┐ ┌───────────┐ ┌────────┐│
  │  │ HTTP / UI │ │ Scheduler │ │Executor││
  │  └───────────┘ └───────────┘ └────────┘│
  │  File-based storage (logs, state, queue)│
  └─────────────────────────────────────────┘

Distributed:

  ┌────────────┐                   ┌────────────┐
  │ Scheduler  │                   │ HTTP / UI  │
  │            │                   │            │
  │ ┌────────┐ │                   └─────┬──────┘
  │ │ Queue  │ │  Dispatch (gRPC)        │
  │ │(file)  │ │─────────┐               │
  │ └────────┘ │         │               │
  └────────────┘         ▼               ▼
                    ┌─────────────────────────┐
                    │      Coordinator        │
                    │  (gRPC task dispatch,   │
                    │   worker registry,      │
                    │   health monitoring)    │
                    └────────▲────────────────┘
                             │
                   Worker poll / task response
                   Heartbeat / ReportStatus /
                   StreamLogs (gRPC)
                             │
               ┌─────────────┴─────────────┐
               │             │             │
          ┌────┴───┐    ┌────┴───┐    ┌────┴───┐
          │Worker 1│    │Worker 2│    │Worker N│
          └────────┘    └────────┘    └────────┘

Quick Start ​

Install ​

curl -fsSL https://raw.githubusercontent.com/dagucloud/dagu/main/scripts/installer.sh | bash

irm https://raw.githubusercontent.com/dagucloud/dagu/main/scripts/installer.ps1 | iex

docker run --rm -v ~/.dagu:/var/lib/dagu -p 8080:8080 ghcr.io/dagucloud/dagu:latest dagu start-all

brew install dagu

helm repo add dagu https://dagucloud.github.io/dagu
helm repo update
helm install dagu dagu/dagu --set persistence.storageClass=<your-rwx-storage-class>

The script installers run a guided wizard that installs Dagu, adds it to your PATH, sets up a background service, and creates the initial admin account. Homebrew, Docker, and Helm install without the wizard. See the Installation Guide for all options.

Create and Run a Workflow ​

cat > hello.yaml << 'EOF'
steps:
  - command: echo "Hello from Dagu!"
  - command: echo "Step 2"
EOF

dagu start hello.yaml

Start the Server ​

dagu start-all

Visit http://localhost:8080

Built-in Step Types ​

Common built-in step types include:

DAGs can also declare reusable step_types that expand to builtin step types at load time. See Custom Step Types and Step Types for the exact configuration surface.

Scheduling and Reliability ​

Security and Access Control ​

Authentication ​

Four authentication modes, configured via DAGU_AUTH_MODE:

Role-Based Access Control ​

When using builtin auth, five roles control access:

API keys can be created with independent role assignments. Audit logging tracks all actions.

TLS and Secrets ​

• TLS for the HTTP server (DAGU_CERT_FILE, DAGU_KEY_FILE)
• Mutual TLS for gRPC coordinator/worker communication (DAGU_PEER_CERT_FILE, DAGU_PEER_KEY_FILE, DAGU_PEER_CLIENT_CA_FILE)
• Secret management with three providers: environment variables, files, and HashiCorp Vault

Observability ​

Prometheus Metrics ​

Dagu exposes Prometheus-compatible metrics at the /metrics endpoint:

OpenTelemetry ​

Per-DAG OpenTelemetry tracing configuration with OTLP endpoint, custom headers, resource attributes, and TLS options.

Structured Logging and Notifications ​

• JSON or text format logging (DAGU_LOG_FORMAT), per-run log files with separate stdout/stderr capture per step
• Slack and Telegram bot integration for run status events (succeeded, failed, aborted, waiting, rejected)
• Email notifications on DAG success, failure, or wait status via SMTP
• Per-DAG webhook endpoints with token authentication

Distributed Execution ​

The coordinator/worker architecture distributes DAG execution across multiple machines:

• Coordinator: gRPC server managing task distribution, worker registry, and health monitoring
• Workers: Connect to the coordinator, pull tasks via long-polling, execute DAGs locally, stream logs back
• Worker labels: Route DAGs to specific workers based on labels (e.g., gpu=true, region=us-east-1)
• Health checks: HTTP health endpoints on coordinator and workers for load balancer integration
• Queue system: File-based persistent queue with configurable concurrency limits

# Start coordinator
dagu coord

# Start workers (on separate machines)
DAGU_WORKER_LABELS=gpu=true,memory=64G dagu worker

See the Distributed Execution documentation for setup details.

Workflow Examples ​

Parallel Execution with Dependencies ​

type: graph
steps:
  - id: extract
    command: ./extract.sh

  - id: transform_a
    command: ./transform_a.sh
    depends: [extract]

  - id: transform_b
    command: ./transform_b.sh
    depends: [extract]

  - id: load
    command: ./load.sh
    depends: [transform_a, transform_b]

Docker Step ​

steps:
  - name: build
    container:
      image: node:20-alpine
    command: npm run build

Retry with Exponential Backoff ​

steps:
  - name: flaky-api-call
    command: curl -f https://api.example.com/data
    retry_policy:
      limit: 3
      interval_sec: 10
      backoff: 2
      max_interval_sec: 120
    continue_on:
      failure: true

Scheduling with Overlap Control ​

schedule:
  - "0 */6 * * *"
overlap_policy: skip
timeout_sec: 3600
handler_on:
  failure:
    command: notify-team.sh
  exit:
    command: cleanup.sh

Sub-DAG Composition ​

steps:
  - name: extract
    call: etl/extract
    params: "SOURCE=s3://bucket/data.csv"

  - name: transform
    call: etl/transform
    params: "INPUT=${extract.outputs.result}"
    depends: [extract]

  - name: load
    call: etl/load
    params: "DATA=${transform.outputs.result}"
    depends: [transform]

SSH Remote Execution ​

steps:
  - name: deploy
    type: ssh
    with:
      host: prod-server.example.com
      user: deploy
      key: ~/.ssh/id_rsa
    command: cd /var/www && git pull && systemctl restart app

See Examples for more patterns.

Version-Controlled Workflows ​

Dagu supports Git sync to keep DAG definitions, agent markdown files, and managed documents version-controlled. Enable DAGU_GITSYNC_ENABLED=true with a repository URL, and Dagu pulls tracked files from a Git branch. Optional auto-sync polls the repository at a configurable interval (default 300s). Supports token and SSH authentication.

See Git Sync for configuration.

CLI Reference ​

Full CLI and environment variable reference: CLI | Configuration Reference

Learn More ​

Community ​