Variables & Parameters Reference
DataKnits provides a robust variable resolution engine to parameterize pipelines, manage credentials securely, and dynamically inject environment and runtime contexts across execution environments.
Platform Scoping
Variables allow design-time templates to adjust to runtime contexts dynamically. This ensures that generated PySpark or SQL code remains clean, environment-agnostic, and safe for execution in highly regulated enterprise ecosystems.
Interactive Variable Simulator
Test the resolution engine using our dynamic playground. Select a simulated runtime environment context, type expressions containing variables, and witness how the engine resolves values and automatically enforces security masking for credentials.
Click variables on the left to copy them, or write your own template expressions on the right.
| Token | Value |
|---|
Scope & Resolution Order
DataKnits uses a hierarchical priority system to resolve conflicting variable declarations. Variables with closer scopes override broader ones. Below is the official resolution chain from highest priority to lowest priority:
| Priority | Scope | Source / Provider | Target Coverage & Purpose |
|---|---|---|---|
| 1 (Highest) | Secret Vaults | AWS Secrets Manager, Azure Key Vault, HashiCorp Vault | Injects connection passwords, tokens, API keys securely at runtime. Never saved in plaintext metadata. |
| 2 | Runtime Injection | Orchestration triggers, webhook payloads, CLI arguments | Dynamic execution inputs (e.g. ${runtime.INPUT_FILE_NAME}) sent during triggers. |
| 3 | Environment Variables | System variables (e.g. ${env.DB_PASSWORD}) |
Configures database connection parameters distinct per deployment environment (Dev/Test/Prod). |
| 4 | Pipeline variables | Visual designer config parameters | Restricted to a single PySpark/SQL pipeline DAG lifecycle. |
| 5 | Project Variables | Project attributes & configs | Shared variables across all pipelines under a specific project folder hierarchy. |
| 6 | Global Variables | Organization level parameters | Default parameters applicable platform-wide across all organizations. |
| 7 (Lowest) | System Variables | DataKnits Agent execution framework | Pre-defined internal metadata, such as ${sys.RUN_DATE}, ${sys.JOB_ID}, and ${sys.COMPILER_VERSION}. |
Scenarios & Requirements Catalog
This section documents all 500 validated enterprise usage scenarios alongside the 10 critical advanced requirements recommended in our architectural audits. Use the search input and categories filter below to explore scenarios instantly.