Pipeline Specification

Define reusable, deterministic data transformation workflows.

Manifest Format

Pipelines are defined as JSON manifests with the following structure:

{
  "name": "pipeline-name",
  "version": "1.0.0",
  "description": "What this pipeline does",
  "steps": [
    {
      "operator": "namespace.operation",
      "params": { ... }
    }
  ]
}

Required Fields

Field	Type	Description
`name`	string	Pipeline identifier (kebab-case)
`version`	string	Semantic version (x.y.z)
`steps`	array	Ordered list of operations

Step Structure

Field	Type	Required	Description
`operator`	string	Yes	Operator name (e.g., `json.parse`)
`params`	object	No	Operator parameters
`id`	string	No	Step identifier for debugging
`comment`	string	No	Documentation comment

Execution Model

Sequential: Steps execute in order, each receiving output from previous step
Deterministic: Same input always produces same output
Fail-fast: Pipeline stops on first error
Type-safe: Input/output types are validated between steps

Running Pipelines

CLI

# From file
json-toolbox run pipeline.json < input.json

# From stdin (use - for manifest)
echo '{"steps":[{"operator":"json.format"}]}' | json-toolbox run -

Browser

Use the Pipeline tab in the web app to build and run pipelines visually.

Example: Multi-step Pipeline

{
  "name": "api-response-transform",
  "version": "1.0.0",
  "description": "Extract and flatten nested API response",
  "steps": [
    {
      "id": "parse",
      "operator": "json.parse",
      "comment": "Parse JSON input"
    },
    {
      "id": "extract",
      "operator": "query.jsonpath",
      "params": { "path": "$.data.items[*]" },
      "comment": "Extract items array"
    },
    {
      "id": "filter",
      "operator": "transform.filter",
      "params": { "key": "active", "operator": "eq", "value": true },
      "comment": "Keep only active items"
    },
    {
      "id": "format",
      "operator": "json.stringify",
      "params": { "indent": 2 },
      "comment": "Pretty print output"
    }
  ]
}

Presets

Built-in presets are available in cli/presets/:

csv-to-json.json - Parse CSV to JSON array
json-to-yaml.json - Convert JSON to YAML
xml-to-json.json - Parse XML to JSON
data-cleanup.json - Repair and dedupe
aggregate-stats.json - Calculate statistics
ndjson-to-json.json - Parse NDJSON to array

json-toolbox run presets/csv-to-json.json < data.csv

Validation

Manifests can be validated against the JSON Schema:

json-toolbox validate-manifest pipeline.json

Schema location: cli/dist/manifest-schema.json