Platform Architecture

DataKnits separates the Control Plane (design, compile, govern) from the Execution Plane (run on your Spark). Your data never transits through DataKnits servers — only generated code artifacts and metadata do.

Logical Architecture

DataKnits follows a layered, decoupled architecture:

Control Plane Components

• Web UI (React/TypeScript) — Workspace Router, Multi-Tab Canvas, Recursive Mapping Builder, Env/Secrets Management, Execution Monitor.
• Backend API (Node.js/TypeScript) — AuthN/AuthZ (JWT + RBAC), Pipeline Service (versioning/IR), Codegen Service (TS Registry), Execution Service (Log Classifier), Strategy Planner (Auto-Routing), Secret Vault (pgcrypto).
• PostgreSQL Cluster — Users & Attributes, Project Hierarchy (LTREE), Pipeline Content (JSONB IR), Connector Registry, Execution Plane (Runs/Logs), Audit & History Triggers.

Execution Plane

The Execution Plane is entirely on customer-owned infrastructure. DataKnits generates a self-contained PySpark script (or SQL for pushdown scenarios), which is submitted to:

• Remote Spark via spark-submit — EMR, YARN, on-prem clusters.
• Databricks Jobs API — Delta Lake workloads on Databricks.
• Google Dataproc — Managed Spark on GCP.
• Local/Embedded Spark — Development and small automation tasks.

Compute Modes

Pushdown Eligibility System

DataKnits automatically analyzes each pipeline segment and determines whether it can be pushed down to the source database or must execute in PySpark. Rules:

• Eligible for pushdown: Single-source segments where all transformation functions are natively supported by the source engine.
• Forced to PySpark: Cross-source JOINs, PySpark-derived column references, unsupported functions in the source engine.
• Users can toggle execution points manually in the pipeline canvas. The system enforces hard locks when a switch would cause data loss or incorrect results.

Connector Architecture

All connectors — regardless of technology — expose a uniform interface:

• connect / test
• listDatabases / listSchemas / listTables / describeTable
• previewData (Apache Arrow IPC, session-scoped encryption)
• generateSparkReadConfig / generateSparkWriteConfig

Credential Zero-Knowledge

Credentials are stored only as encrypted payloads (AES-256 via pgcrypto). Decryption happens at pipeline execution time in the Spark context. The secret resolution chain at runtime is:

1. Spark session config (injected by DataKnits codegen)
2. AWS Secrets Manager
3. Azure Key Vault
4. GCP Secret Manager
5. HashiCorp Vault
6. Environment variable fallback

Pipeline IR & Versioning

Pipeline versions are immutable. The UI emits a stable JSONB Intermediate Representation (IR) that is independent of UI widget structure. The IR is stored in PostgreSQL (pipeline_content table).

• Edits create new versions or drafts — existing versions are never mutated.
• DAG load target: < 2 seconds for 300-node pipelines.
• Validation target: < 1 second.
• Compile target: < 30 seconds typical; large pipelines < 3 minutes.

On-Prem Iceberg Lakehouse (Docker)

DataKnits ships a reference on-prem lakehouse via Docker Compose with four services:

Data flow: Spark ↔ Iceberg catalog (alias lakehouse) ↔ MinIO (s3://lakehouse/warehouse). Persistence comes from the Docker volume compose_minio-data.

Non-Functional Requirements