Code Migration & Deployment

How to build, package, and deploy DataKnits to Azure App Service, set up the Docker-based on-prem lakehouse, configure CI/CD, and migrate pipelines from legacy ETL tools.

On this page

Deploy to Azure App Service
Docker Compose Setup
CI/CD Pipeline
Migrating from Legacy ETL Tools
Runtime Dependencies

Deploy to Azure App Service

Target environment: Resource group rde-app-group, region South India, App Service RDE (legacy Azure resource name — cannot be renamed in place). Runtime: NODE|24-lts.
Public URL: https://etl.dataknits.com

Prerequisites

Azure CLI logged in: az login. Verify with az account show.
Node 24 installed locally (matches App Service runtime).
Working tree on the commit to deploy.

Step 1 — Frontend Build (Vite)

cd Frontend
rm -rf dist
npm run build

Outputs Frontend/dist/ containing index.html plus hashed assets in assets/.

Step 2 — Backend Build (TypeScript)

cd ../Backend
rm -rf dist
npm run build

This runs: tsc → tsc-alias (path rewriting) → copies SQL migrations to dist/db/migrations/.

Step 3 — Wire Frontend into Backend Static Root

rm -rf Backend/dist/public
cp -r Frontend/dist Backend/dist/public

The Node server serves dist/public/ when SERVE_FRONTEND=true.

Step 4 — Package the Deployment ZIP

cd Backend
TS=$(date +%Y%m%d%H%M%S)
ZIP="../etl1-azure-deploy-${TS}.zip"
zip -rq1 -X "$ZIP" dist src/db/migrations node_modules package.json

Step 5 — Deploy to Azure

az webapp deployment source config-zip \
  --resource-group rde-app-group \
  --name RDE \
  --src "$ZIP"

Step 6 — Set Startup Command

az webapp config set \
  --resource-group rde-app-group \
  --name RDE \
  --startup-file "node dist/api/server.js"

Step 7 — Configure App Settings

Set all required environment variables (from Admin Guide → Environment Variables) via the Azure portal or:

az webapp config appsettings set \
  --resource-group rde-app-group \
  --name RDE \
  --settings DB_HOST="..." DB_NAME="etl_db" JWT_SECRET="..." ...

Docker Compose Setup

For on-premises deployments or development environments, DataKnits ships a Docker Compose configuration that provisions the full stack including the Iceberg Lakehouse.

Services in docker-compose.etl.yml

Service	Image	Purpose
`etl1-api`	DataKnits Backend	Core API server and code generation engine.
`etl1-db`	`postgres:15`	Metadata database with pgcrypto, ltree, pg_trgm extensions.
`etl1-spark-iceberg`	Apache Spark + Iceberg	PySpark execution engine for local/dev pipelines.
`etl1-iceberg-rest`	Iceberg REST Catalog	Iceberg table catalog (namespaces, schemas, partitions).
`etl1-minio`	MinIO	S3-compatible object storage for lakehouse warehouse files.

Start the Stack

./start.sh
# or manually:
docker compose -f docker-compose.etl.yml up -d

Stop the Stack

./stop.sh
# or:
docker compose -f docker-compose.etl.yml down

CI/CD Pipeline

The recommended CI/CD flow via Azure DevOps or GitHub Actions:

Lint & Type-check — tsc --noEmit on both Frontend and Backend.
Unit Tests — npm test in both packages.
Build — Run steps 1–4 from the Azure deployment section above.
Deploy to Staging — Zip-deploy to the staging App Service slot.
Smoke Tests — Run Playwright E2E tests against staging.
Swap Slots — Azure slot swap from staging to production (zero-downtime).

Migrating from Legacy ETL Tools

DataKnits is designed for teams migrating from tools like Informatica, Talend, SSIS, or hand-written PySpark scripts.

Migration Patterns

Source Tool	Approach
Informatica / Talend	Map each mapping/component to a DataKnits node type. Source/Target connectors map 1:1. Transformations map to DataKnits nodes (Filter, Join, Aggregate, Derive). Complex expressions go in Custom SQL nodes.
SSIS (.dtsx)	Parse SSIS package XML to extract source/target connections and data flow tasks. Re-create in DataKnits with equivalent nodes. SSIS Derived Column → DataKnits Derive Node.
Hand-written PySpark	Reverse-engineer logic into DataKnits nodes. Complex custom logic stays in Custom SQL or Derive nodes. DataKnits can also import and run existing `.py` scripts as-is via the Execution Service.
SQL stored procedures	Wrap in Custom SQL node or break down into Aggregate + Filter + Join nodes for visual representation. DataKnits generates equivalent SQL for pushdown scenarios.

What Carries Over Automatically

JDBC connection strings → DataKnits JDBC connector.
S3 / Azure Blob / GCS paths → DataKnits Cloud Storage connectors.
Table schema metadata → imported via DataKnits Metadata Catalog.

Runtime Dependencies

DataKnits does not bundle Spark JARs — they must be on the classpath of your Spark cluster at execution time.

Key Backend NPM Packages

Package	Version	Purpose
`express`	^4.18.2	HTTP server and REST API routing.
`pg`	^8.11.3	PostgreSQL client.
`uuid`	^9.0.1	RFC-4122 UUID generation.
`winston`	^3.11.0	Structured JSON logging.
`typescript`	^5.3.0	TypeScript compiler (dev).

Required PostgreSQL Extensions

Extension	Purpose
`pgcrypto`	AES-256 encryption for connector secrets and SSH tunnel configs.
`ltree`	Hierarchical path storage for folder/project trees.
`pg_trgm`	Trigram-based fuzzy text search for the metadata catalog.

Required Spark JARs (examples)

JAR	Connector
`postgresql.jar`	PostgreSQL, Greenplum
`ojdbc11.jar`	Oracle, OCI Autonomous DB
`mssql-jdbc.jar`	SQL Server
`snowflake-jdbc.jar`	Snowflake
`hadoop-aws.jar` + `aws-java-sdk-bundle.jar`	Amazon S3
`gcs-connector-hadoop3-shaded.jar`	Google Cloud Storage
`hadoop-azure.jar` + `azure-storage.jar`	Azure Blob Storage, ADLS Gen2

See Technologies → individual connector pages for the complete JAR list per technology.