File Operations

Move, copy, archive, and clean up files as part of an automated workflow — without writing a pipeline. File Operation, Custom Command, and Condition steps let you orchestrate file handling (FTP/SFTP servers, cloud object storage, HDFS) alongside your pipelines and child orchestrators, all from the same drag-and-drop DAG editor.

Overview

File Operations adds three new step types to the Orchestrator DAG editor, alongside the existing Pipeline, Orchestrator, and Parallel Group steps:

• File Operation — list, download, upload, delete, move, rename, zip, or unzip files on a connected file store.
• Custom Command — run a shell command, either on the DataKnits host, over SSH on a remote server, or through a cloud CLI (AWS/Azure/GCP/OCI/HDFS).
• Condition — evaluate a true/false expression (for example, "did any files arrive?") and route the orchestrator down a success or failure branch accordingly.

These steps live entirely inside orchestrators. They are not available on the pipeline canvas — pipelines remain focused on data transformation, while orchestrators sequence pipelines, sub-orchestrators, and now file/command housekeeping into a single automated flow.

File Storage Connections

File Operation steps run against a Connection you already have configured (the same connections used elsewhere in DataKnits). Supported connection types include:

When you select a connection inside a File Operation step, the panel shows its native method and supported capabilities — this tells you upfront which operations that particular connection can perform before you configure the rest of the step.

Adding Steps to an Orchestrator

1. Open an existing orchestrator (or create a new one) and go to its Editor tab.
2. In the toolbar above the canvas, click Add File Op, Add Custom Command, or Add Condition.
3. A new step card appears on the canvas. Click it to open the configuration panel on the right.
4. Fill in the step's fields (covered below), then drag the green (Success) or red (Failure) connector on the right edge of the step to another step to wire up the flow.
5. Click Save DAG when you're done.

File Operation Step

Click a File Operation step to configure it in the right-hand panel:

Operations Reference

Execution Modes

By default, each connection uses its natural transport method automatically. Override this only when your environment requires a specific transport:

Custom Command Step

Use a Custom Command step for anything outside the standard file operations — running a legacy shell script, invoking a third-party CLI, or performing custom cleanup logic.

Condition Step

A Condition step evaluates a boolean expression against the orchestrator's current parameter values. If the expression is true, the flow follows the step's Success route; if false, it follows the Failure route.

Using Parameters (${param.NAME})

Every path, command body, and expression field in these steps supports ${param.NAME} substitution, resolved at run time against the orchestrator's declared parameters.

• Click the Insert → … button under any supported field to search and insert a parameter token.
• You can also drag a parameter directly from the left panel and drop it into any field.
• Any step's Output Param becomes available to every step that runs after it, so a LIST step's file count, or a Custom Command's exit code/stdout, can drive a later Condition step or be referenced in a later path.

Routing, Retries & Alerts

File Operation, Custom Command, and Condition steps use the same routing model as pipeline and orchestrator steps:

• Success / Failure routes — drag from the green or red connector on a step to wire it to the next step. Leave a route unset to fall through to "Next Step" (success) or "Stop" (failure), or explicitly choose End / Continue.
• Retry Count / Retry Delay (sec) — automatically retry a failed step before routing to Failure.
• Timeout (min) — maximum time the step is allowed to run.
• Alerts — enable Alert on Start, Alert on Success, and/or Alert on Failure per step to trigger the notification rules configured under Settings → Email.

Worked Example: Watch, Ingest, Archive

1. Add a File Operation step: Operation LIST, Connection = your SFTP connection, Source Path /incoming/${param.LOAD_DATE}/, Output Param FILE_COUNT.
2. Add a Condition step: Expression ${param.FILE_COUNT} > 0. Route its Success edge to the pipeline that ingests the data.
3. After the pipeline's Success route, add a second File Operation step: Operation MOVE, Source Path /incoming/${param.LOAD_DATE}/orders.csv, Destination Path /archive/${param.LOAD_DATE}/orders.csv.
4. Optionally add a Custom Command step afterward to clean up any temporary local files.
5. Save the DAG and attach a schedule under the orchestrator's Schedule tab.

Troubleshooting