Getting Started
Build your first pipeline in under 5 minutes
Connect a Source
Go to Settings → Connections and add a database, cloud bucket, or file server. DataKnits supports 40+ technologies out of the box.
Browse all connectors →Browse the Catalog
Use the Metadata Catalog to find tables and columns. Use Data Preview (100-row sample) to confirm your data before designing the pipeline.
Create a Project
Projects are the logical container for your pipelines, connections, and metadata. Go to the Projects tab and click New Project.
Admin: Projects guide →Design the Pipeline
Drag a Source Node, chain Transform Nodes (Filter, Join, Aggregate, SCD), and connect a Target Node onto the canvas.
Canvas reference →Validate & Run
Click Validate to check for errors (DAG load < 2s), then Run to execute. DataKnits generates native PySpark or SQL — no black box.
Execution modes →Core Platform Concepts
Control vs Execution Plane
The Control Plane (DataKnits server) stores metadata, validates pipelines, and compiles artifacts. The Execution Plane runs generated PySpark jobs on your own Spark infrastructure — your data never transits through DataKnits.
Code Generation
Every pipeline compiles to portable, readable PySpark (or SQL for pushdown scenarios). You can inspect, download, and run the generated code independently — zero runtime vendor lock-in.
Secrets & Security
Credentials are encrypted with AES-256 (pgcrypto). Secrets are never embedded in generated code — they are resolved at runtime via Spark session config, AWS Secrets Manager, Azure Key Vault, or GCP Secret Manager.
RBAC & Multi-Tenancy
Permissions cascade through Organization → Workspace → Project → Folder → Pipeline. Roles (Super Admin, Org Admin, Developer, Analyst, Operator, Approver) enforce least-privilege access at every layer.
Full RBAC reference →Knowledge Hub Sections
Architecture
Logical architecture, compute modes, pipeline IR, deployment topology, performance SLAs, and on-prem Lakehouse wiring.
- Control vs Execution Plane
- Compute Modes (Spark, SQL Pushdown, Databricks)
- Iceberg Lakehouse on Docker
- Non-functional requirements
Admin Guide
Platform-wide settings, organization setup, workspace management, environment configuration, connections, and audit trails.
- Organization & Workspace setup
- Connection management
- Environment variables
- Audit logging & compliance
Development
Canvas usage, transformation nodes, PySpark code generation, pushdown strategies, validation framework, and keyboard shortcuts.
- Pipeline Canvas & Node Types
- All 39 Transformation Components
- Pushdown Eligibility System
- SCD Types 1, 2, 3 & 4
Code Migration & Deployment
Azure App Service deployment, Docker setup, environment configuration, CI/CD pipelines, and migration from legacy ETL tools.
- Deploy to Azure App Service
- Docker Compose setup
- Build & package pipeline
- Legacy ETL migration patterns
User Management
Role definitions, permission matrices, RBAC inheritance, user lifecycle CRUD, team groups, delegation, and resource-level access.
- System & Functional Roles
- Full Permission Matrix
- Resource Hierarchy (Org→Pipeline)
- Teams, Groups & Delegation
SSO & Authentication
JWT-based auth, Single Sign-On (SAML 2.0 / OIDC), MFA, session management, IP allowlists, and password policy configuration.
- JWT & Session Management
- SAML 2.0 / OIDC Integration
- MFA Enforcement
- IP Allowlist & Password Policies
Integrations
Connect external tools to DataKnits. Step-by-step setup for the Jira issue tracker integration.
- Generate a Jira API Token
- Find Your Project Key
- Configure Settings → Integrations
- Raise an Issue & Troubleshooting
File Operations
List, transfer, archive, and clean up files across FTP/SFTP, cloud storage, and HDFS directly from the Orchestrator DAG editor.
- File Operation Steps (GET/PUT/DELETE/MOVE/RENAME/ZIP/UNZIP/LIST)
- Custom Command Steps (OS / SSH / Cloud CLI)
- Condition Steps & Parameter Substitution
- Retries, Routing & Alerts
Secure Agent
Connect on-premises data sources to DataKnits with an outbound-only agent — no inbound firewall rules, ever.
- How the Secure Agent Works
- Provisioning & Installing an Agent
- Running Jobs Through an Agent
- Lifecycle: Deactivate, Activate, Revoke
Technologies Supported
40+ connectors across 8 categories. Each technology has its own reference page covering connection fields, dependencies, and known limitations.
- Relational Databases (PostgreSQL, Oracle, SQL Server…)
- Cloud Warehouses (Snowflake, Databricks, BigQuery…)
- Streaming (Kafka, IBM MQ, Kinesis, Pub/Sub…)
- NoSQL, Object Storage, Time-Series & more
Platform Resource Hierarchy
Can't find what you're looking for?
Our engineering team is ready to help with custom integration questions, enterprise deployments, and architecture reviews.