Google Cloud Pub/Sub as a streaming source or publish target. Unbounded, subscription-based consumption with a validated ETL-native runtime path and a bounded-only PySpark path. Because live preview would otherwise consume production messages, schema discovery uses two dedicated, permission-scoped mechanisms — see Learning a Message's Shape.
Connection Fields
Source: GcpPubSubConnectorPlugin.ts, client construction in gcp/pubSubClient.ts (connector type GCP_PUBSUB). Unlike the GCS/BigQuery/Bigtable plugins, Pub/Sub uses the official @google-cloud/pubsub client library rather than raw REST, because it needs the client's streaming-pull machinery.
| Field | Required | Default | Notes |
|---|---|---|---|
project_id | Yes | — | |
auth_method | Yes | APPLICATION_DEFAULT | ATTACHED_SERVICE_ACCOUNT · WORKLOAD_IDENTITY_FEDERATION · SERVICE_ACCOUNT_IMPERSONATION · APPLICATION_DEFAULT · SERVICE_ACCOUNT_KEY. Recommended: WORKLOAD_IDENTITY_FEDERATION. |
resource_project_id | No | — | Set if the topics/subscriptions live in a different project than project_id. |
quota_project_id | No | — | Project billed for API quota. |
impersonated_service_account | For SERVICE_ACCOUNT_IMPERSONATION | — | Target service account email. |
impersonation_delegates | No | — | Comma-separated delegation chain. |
impersonation_lifetime_seconds | No | 3600 | Range 600–3600. |
endpoint_mode | Yes | — | GLOBAL · REGIONAL · LOCATIONAL · EMULATOR · CUSTOM. |
endpoint_host | For non-GLOBAL modes | — | |
tls_validation | Yes | — | Keep true for every real Google endpoint. |
request_timeout_ms | No | 15000 | |
test_topic_id / test_subscription_id | No | — | Optional — lets Test Connection verify exact publish/consume IAM instead of a coarser discovery-level check. |
ack_deadline_seconds | No | 60 | Range 10–600. |
max_outstanding_messages / max_outstanding_bytes | No | 1000 / 100MB | Subscriber flow control. |
enable_message_ordering | No | false | |
batch_max_messages / batch_max_bytes / batch_max_latency_ms | No | 100 / 1MB / 10ms | Publisher batching. |
service_account_key_json (secret) | For SERVICE_ACCOUNT_KEY | — |
Authentication Methods
Five auth methods are supported. WORKLOAD_IDENTITY_FEDERATION is recommended for production; APPLICATION_DEFAULT is the connector default, intended for local/on-prem backend deployments rather than production use.
For SERVICE_ACCOUNT_IMPERSONATION, the calling principal only needs roles/iam.serviceAccountTokenCreator on the target service account — not the target account's own data-plane roles.
Required IAM Roles
Grant only what a given pipeline role needs:
- Consume a subscription (source node, or the Infer Schema button — see below):
roles/pubsub.subscriber, which grantspubsub.subscriptions.consume. - Publish (sink node):
roles/pubsub.publisher, which grantspubsub.topics.publish. - Browse topics, subscriptions, snapshots, and schemas in the metadata tree:
roles/pubsub.viewer. SERVICE_ACCOUNT_IMPERSONATION: the calling principal needsroles/iam.serviceAccountTokenCreatoron the target service account.
Test Connection Behaviour
If test_topic_id / test_subscription_id are configured, Test Connection performs a real, non-mutating testPermissions() IAM check for pubsub.topics.publish and pubsub.subscriptions.consume. Without a test resource configured, it falls back to a real (but coarser) topic/subscription discovery probe. Either way it is a genuine network + auth check, not a hardcoded pass.
Runtime Paths: ETL-Native vs PySpark
| Path | Status | Guarantees |
|---|---|---|
| ETL-Native (default, validated) |
Primary path | Durable per-message dedup ledger (Postgres), encrypted quarantine, exactly-once ack support for BUSINESS_EVENT_ID dedup mode. Uses gcpPubSubDrainGenerator in the runtime's source-streams module. |
| PySpark (bounded pull only) |
Bounded pipelines only | Drives the official google-cloud-pubsub Python client directly. No dedup ledger, no quarantine. A crash between DataFrame construction and downstream sink commit can redeliver duplicates on the next run. |
The codegen/UI gate only blocks runtimeMode: CONTINUOUS Pub/Sub pipelines from generating PySpark (surfaced as a PUBSUB_PYSPARK_CONTINUOUS_UNSUPPORTED error) — bounded Pub/Sub pipelines can generate PySpark today, with the caveats above surfaced at codegen time via a PUBSUB_PYSPARK_BOUNDED_ONLY warning.
Building a Pub/Sub → Parse JSON → BigQuery Pipeline
- In Connections, create a
GCP_PUBSUBconnection (project ID, auth method, plus either the service account key secret or an on-GCP-resolvable auth method), then create aGCP_BIGQUERYconnection (project + dataset). Run Test Connection on each — both perform real network/auth/authz checks. - Start a New Pipeline, drag a Source node onto the canvas, pick the Pub/Sub connection, and pick the subscription from the metadata browser.
- Drag a Parse JSON Column node after it to flatten the raw message body into typed columns — see Learning a Message's Shape for how to populate its output schema.
- Drag a Target node, pick the BigQuery connection, pick or create the destination table, and map columns — now real, since the Parse JSON node has a concrete output schema.
- Run with strategy AUTO or Force ETL Native. ETL-Native has full support for the Parse JSON Column transform and is the validated, dedup-ledger-backed path for Pub/Sub.
Learning a Message's Shape
Pub/Sub sources block live data preview by design, to avoid consuming or leasing production messages. To populate a Parse JSON Column node's output schema before mapping columns downstream, use one of two mechanisms:
1. Infer Schema (pulls one real message)
Calls the node preview API, which for a Pub/Sub source routes to a low-level SubscriberClient.pull() for exactly one message, then immediately calls modifyAckDeadline({ ackDeadlineSeconds: 0 }) — the standard synchronous-pull "nack." The message is never acknowledged or consumed and becomes redeliverable right away.
- Permission required:
pubsub.subscriptions.consume(roles/pubsub.subscriber) on the subscription — the same permission a real pipeline run needs. - Real side effect: the message is leased for the brief pull-then-nack window. Redelivery isn't instantaneous — on an exactly-once or ordering-key subscription this can cause a short visibility delay or, rarely, a duplicate-delivery retry. This is a deliberate, bounded trade-off for schema visibility, not a bug: it relaxes the platform's normal "preview never touches production messages" guarantee specifically for this one action.
2. Parse Sample (paste a message body)
Paste a real message body as text. It's parsed entirely client-side using the identical flatten/array-handling logic the backend uses, kept in sync with the server-side implementation.
- Permission required: none. Use this when the connection can't yet consume (IAM not granted yet) or when you'd rather not touch the live subscription at all.
Known Limitations
- PySpark source is bounded-only.
runtimeMode: CONTINUOUShas no PySpark code generation — use ETL-Native for continuous streaming Pub/Sub pipelines. - PySpark source has no durable dedup ledger, quarantine, or exactly-once ack — unlike the ETL-Native path. Use ETL-Native whenever duplicate-delivery guarantees matter.
- Continuous/streaming run-lifecycle management (pause/resume, long-running supervision) is a tracked, unimplemented gap regardless of engine.