Pipeline Orchestration

Pipeline orchestration enables agentful to run AI agents in complex, long-running workflows with parallel execution, dependencies, and robust error handling.

What are Pipelines?

Pipelines are YAML-based workflow definitions that orchestrate multiple AI agents to complete complex tasks. Think of them as "GitHub Actions for AI agents."

Key Features

Async Execution - Long-running workflows (up to 30 minutes)
Parallel Jobs - Execute multiple agents concurrently
Dependency Management - Jobs wait for dependencies to complete
Conditional Execution - Skip jobs based on conditions
Retry Policies - Automatic retry with exponential backoff
State Persistence - Resume interrupted pipelines
Progress Tracking - Real-time progress updates
CI/CD Integration - Export to GitHub Actions, GitLab CI, Jenkins

Quick Start

1. Create a Pipeline

Create .agentful/pipelines/my-feature.yml:

name: my-feature
version: 1.0
 
jobs:
  - id: implement-backend
    agent: backend
    task: Implement authentication API
 
  - id: implement-frontend
    agent: frontend
    dependsOn: implement-backend
    task: Create login UI
 
  - id: test
    agent: tester
    dependsOn:
      - implement-backend
      - implement-frontend
    task: Write tests for authentication

2. Run the Pipeline

npx agentful pipeline run --pipeline .agentful/pipelines/my-feature.yml

3. Check Status

npx agentful pipeline status --run-id my-feature-1234567890-abc

When to Use Pipelines

Scenario	Use Pipeline?	Why
Complete feature (backend + frontend + tests)	✅ Yes	Multiple dependent agents
Quick bug fix	❌ No	Single agent is faster
Production deployment	✅ Yes	Safety checks and validation
Continuous validation (CI)	✅ Yes	Parallel execution for speed
Code review	❌ No	Reviewer agent alone is sufficient
Complex refactoring	✅ Yes	Multiple stages with validation

Pipeline Structure

Basic Pipeline

name: pipeline-name
version: 1.0
description: What this pipeline does
 
jobs:
  - id: job-1
    agent: backend
    task: What the agent should do
 
  - id: job-2
    agent: frontend
    dependsOn: job-1
    task: What the agent should do

Full Pipeline Options

name: advanced-pipeline
version: 1.0
description: Advanced pipeline with all options
 
# Trigger configuration
triggers:
  - type: push
    branches: [main]
  - type: pull_request
  - type: schedule
    cron: "0 0 * * *"
  - type: manual
 
# Environment variables
env:
  NODE_ENV: production
  FEATURE_FLAG: enabled
 
# Concurrency settings
concurrency:
  maxConcurrentJobs: 3
  cancelInProgress: false
 
# Default timeout (30 minutes)
timeout: 1800000
 
jobs:
  - id: job-id
    name: Human-readable job name
    agent: agent-name
    task: Detailed task description
    prompt: Additional instructions
 
    # Dependencies
    dependsOn:
      - job-1
      - job-2
 
    # Conditional execution
    when: "job-1.status == 'completed'"
 
    # Inputs from previous jobs
    inputs:
      data: ${{ job-1.output.result }}
 
    # Timeout (overrides default)
    timeout: 600000 # 10 minutes
 
    # Retry policy
    retry:
      maxAttempts: 2
      backoff: exponential # exponential, linear, fixed
      delayMs: 2000
 
    # Continue even if fails
    continueOnError: false
 
    # Execution method
    execution:
      method: subprocess # subprocess or api
      isolation: shared # shared or isolated

Job Dependencies

Sequential Execution

jobs:
  - id: step-1
    agent: backend
    task: First step
 
  - id: step-2
    agent: frontend
    dependsOn: step-1
    task: Second step (waits for step-1)

Parallel Execution

jobs:
  - id: backend
    agent: backend
    task: Backend work
 
  - id: frontend
    agent: frontend
    task: Frontend work (runs in parallel with backend)
 
  - id: tests
    agent: tester
    dependsOn:
      - backend
      - frontend
    task: Tests (waits for both to complete)

Diamond Dependency

jobs:
  - id: analyze
    agent: architect
    task: Analyze requirements
 
  - id: backend
    agent: backend
    dependsOn: analyze
 
  - id: frontend
    agent: frontend
    dependsOn: analyze
 
  - id: integration
    agent: tester
    dependsOn:
      - backend
      - frontend

Conditional Execution

Skip jobs based on conditions:

jobs:
  - id: quality-check
    agent: reviewer
    task: Run quality gates
 
  - id: fix-issues
    agent: fixer
    dependsOn: quality-check
    when: "quality-check.status == 'failed'"
    task: Fix issues (only runs if quality check failed)

Retry Policies

Handle non-deterministic failures:

jobs:
  - id: api-call
    agent: backend
    task: Call external API
 
    retry:
      maxAttempts: 3
      backoff: exponential
      delayMs: 2000
    # Retry delays: 2s, 4s, 8s

Backoff Strategies

Exponential: delay * 2^(attempt-1) - Best for API rate limits
Linear: delay * attempt - Good for temporary network issues
Fixed: delay - Simple retry with constant delay

Context Passing

Pass data between jobs:

jobs:
  - id: analyze
    agent: architect
    task: Analyze feature requirements
 
  - id: implement
    agent: backend
    dependsOn: analyze
    task: |
      Implement feature based on analysis
      Architecture: {{analyze.output.architecture}}
      Patterns: {{analyze.output.patterns}}
    inputs:
      requirements: ${{ analyze.output.requirements }}

Progress Tracking

Real-time progress updates:

⏳ Job started: Implement Backend Logic (attempt 1)
   Progress: 25%
   Progress: 50%
   Progress: 75%
✅ Job completed: Implement Backend Logic (18.3s)
 
⏳ Job started: Write Test Coverage (attempt 1)
   Progress: 30%
   Progress: 60%
✅ Job completed: Write Test Coverage (12.5s)

State Persistence

Pipelines automatically save state after each job. If interrupted (crash, timeout, Ctrl+C):

# Resume from where it left off
npx agentful pipeline resume --run-id my-feature-1234567890-abc

State files are stored in .agentful/pipelines/runs/.

Resource Management

Control concurrent execution:

concurrency:
  maxConcurrentJobs: 3
  cancelInProgress: true # Cancel old runs when new one starts

Concurrency Guidelines

Project Size	Recommended Concurrency
Small (<5 agents)	2-3
Medium (5-10 agents)	3-5
Large (>10 agents)	5-7

Higher concurrency = faster completion but higher LLM API costs.

Error Handling

Job-Level Errors

jobs:
  - id: risky-job
    agent: backend
    task: Risky operation
    continueOnError: true # Don't fail pipeline if this job fails
 
  - id: critical-job
    agent: backend
    task: Critical operation
    continueOnError: false # Fail pipeline if this job fails (default)

Pipeline-Level Errors

If a job fails without continueOnError: true:

Running jobs complete
Queued jobs are cancelled
Pipeline status = FAILED
Error details saved to state file

CLI Commands

# Run a pipeline
npx agentful pipeline run --pipeline path/to/pipeline.yml
 
# Run with context
npx agentful pipeline run -p pipeline.yml -c context.json
 
# Check status
npx agentful pipeline status --run-id <run-id>
 
# List all runs
npx agentful pipeline list
 
# Cancel a running pipeline
npx agentful pipeline cancel --run-id <run-id>
 
# Resume an interrupted pipeline
npx agentful pipeline resume --run-id <run-id>
 
# Validate pipeline definition
npx agentful pipeline validate --pipeline pipeline.yml

CI/CD Integration

Export to GitHub Actions

npx agentful pipeline export --pipeline pipeline.yml --format github-actions

Output: .github/workflows/agentful-pipeline.yml

Export to GitLab CI

npx agentful pipeline export --pipeline pipeline.yml --format gitlab-ci

Output: .gitlab-ci.yml

Export to Jenkins

npx agentful pipeline export --pipeline pipeline.yml --format jenkins

Output: Jenkinsfile

Best Practices

1. Use Descriptive Job IDs

❌ Bad:

- id: job1
- id: job2

✅ Good:

- id: implement-auth-backend
- id: implement-auth-frontend

2. Set Appropriate Timeouts

# Quick jobs (type checking, linting)
timeout: 300000 # 5 minutes
 
# Medium jobs (unit tests, small features)
timeout: 900000 # 15 minutes
 
# Long jobs (E2E tests, large features)
timeout: 1800000 # 30 minutes

3. Use Retry for Flaky Operations

- id: api-integration-test
  agent: tester
  task: Test external API integration
  retry:
    maxAttempts: 3
    backoff: exponential

4. Parallelize When Possible

❌ Sequential:

- id: backend
  agent: backend
- id: frontend
  agent: frontend
  dependsOn: backend # Unnecessary dependency

✅ Parallel:

- id: backend
  agent: backend
- id: frontend
  agent: frontend # Runs in parallel

5. Use Conditional Jobs for Optimization

- id: deploy-staging
  agent: orchestrator
  when: "context.environment == 'staging'"
  task: Deploy to staging

Examples

See /examples/pipelines/ for complete examples:

feature-development.yml - Complete feature implementation
continuous-validation.yml - Fast CI validation
deploy-production.yml - Safe production deployment

Next Steps

Getting Started - Start using agentful
Commands Reference - All available commands
Agents Overview - Learn about specialized agents
Remote Execution - Run agents on remote servers