Google Dataproc Serverless | DataKnits Knowledge Hub

Dataproc

Dataproc Serverless is a compute connector — it holds no data of its own. Set it as the parent compute connector on a GCS/BigQuery data connection to submit generated PySpark to GCP's Dataproc Batches API instead of running spark-submit locally.

On this page

Identity Delegation
Connection Fields
Required IAM Roles
Test Connection Behaviour
Known Limitations

Identity Delegation

Source: GcpDataprocServerlessConnectorPlugin.ts, submission logic in dataprocServerlessSubmitter.ts (connector type GCP_DATAPROC_SERVERLESS).

Set this connector as the parent_connector_id on a GCS or BigQuery data connection to enable GCP-identity delegation — no explicit credentials are injected into the batch's Spark configuration for those connections. Non-GCP connections (S3, Azure, JDBC) are never delegatable — their credentials are always injected explicitly into runtimeConfig.properties.

Connection Fields

Field	Required	Default	Notes
`gcp_project_id`	Yes	—
`gcp_region`	Yes	—	Batches should run in the same region as the data.
`dataproc_staging_bucket`	Yes	—	Where the generated `.py` file is uploaded. Bucket name only, no `gs://` prefix.
`auth_method`	Yes	`SERVICE_ACCOUNT_KEY`	`SERVICE_ACCOUNT_KEY` · `WORKLOAD_IDENTITY` (recommended) · `WORKLOAD_IDENTITY_FEDERATION` · `LOCAL_USER_IMPERSONATION`.
`workload_identity_pool_provider`	For WIF	—
`dataproc_submitter_service_account`	For WIF / `LOCAL_USER_IMPERSONATION`	—	The identity actually calling the Dataproc API — distinct from the batch's own runtime identity.
`local_adc_file_path`	No (`LOCAL_USER_IMPERSONATION`)	`~/.config/gcloud/application_default_credentials.json`
`azure_managed_identity_client_id`	No (WIF)	—	Only if using a user-assigned managed identity.
`service_account_email`	No	Project's default Compute Engine SA	The identity the batch job itself runs as.
`dataproc_runtime_version`	No	`2.2`	`2.2` = Spark 3.5 · `2.1` = Spark 3.3 · `1.1`.
`gcp_subnetwork`	No	—	Required for Private Google Access to reach non-GCP sources.
`dataproc_batch_ttl_hours`	No	4	Range 1–24. Auto-cancel guard against runaway cost.
`dataproc_extra_jar_uris`	Conditionally	—	Comma-separated `gs://` JDBC driver JARs; required for any JDBC-writing pipeline.
`service_account_key_json` (secret)	For `SERVICE_ACCOUNT_KEY`	—	Used only by the platform to call the Dataproc API and upload to GCS — never forwarded into the Spark job itself.

Required IAM Roles

Dataproc Serverless has two distinct identities in play — keep them separate when granting roles:

Identity	Purpose	Roles required
Submitter identity (what `auth_method` resolves)	Calls the Dataproc API to create the batch and uploads the generated script	`dataproc.batches.create`/`get` (e.g. `roles/dataproc.editor`) on the project, plus `storage.objects.create` on the staging bucket
Batch runtime identity (`service_account_email`)	The identity actually reading/writing your data inside the Spark job	`roles/dataproc.worker`, plus whatever data-connector roles the pipeline touches (GCS/BigQuery/etc.)

For WIF or LOCAL_USER_IMPERSONATION: the impersonating principal only needs roles/iam.serviceAccountTokenCreator (or roles/iam.workloadIdentityUser for WIF) on dataproc_submitter_service_account — the submitter service account itself holds the real Dataproc/GCS roles.
For Secret Manager-backed connector secrets in production, the batch service account also needs roles/secretmanager.secretAccessor.

Test Connection Behaviour

Test Connection for Dataproc Serverless is config-shape-only — it checks that required fields are present but does not call the Dataproc API. Run an actual pipeline to verify end-to-end auth.

Known Limitations

Test Connection does not verify IAM or network reachability — only field completeness. Auth problems surface at Run time, not at connection setup time.
JDBC-writing pipelines require the correct driver JAR to be pre-uploaded to GCS and referenced via dataproc_extra_jar_uris — this is not handled automatically.