Google BigQuery as a cloud data warehouse source or target. ETL-Native reads/writes via the REST API and the BigQuery Storage Read API; Spark reads/writes go through the spark-bigquery-connector, which requires a GCS staging bucket.
Connection Fields
Source: GcpConnectorPlugin.ts (connector type GCP_BIGQUERY).
| Field | Required | Default | Notes |
|---|---|---|---|
bigquery_project_id | Yes | — | |
bigquery_dataset | No | — | Default dataset for metadata browsing and unqualified table references. |
bigquery_location | No | — | e.g. US, EU. |
bigquery_temp_gcs_bucket | Yes for Spark writes | — | Staging bucket the spark-bigquery-connector requires. ETL-Native source/target does not need it. |
bigquery_use_storage_api | No | true | Uses the BigQuery Storage Read API for reads. |
auth_method | No | SERVICE_ACCOUNT_KEY | SERVICE_ACCOUNT_KEY · WORKLOAD_IDENTITY · APPLICATION_DEFAULT. |
gcp_auth_ref | No | — | Reuse credentials from another GCP connector instead of storing a second copy. |
select_batch_size | No | 50000 | Rows per round trip on read. |
insert_batch_size | No | 10000 | Rows per batch on write. |
service_account_key_json (secret) | For SERVICE_ACCOUNT_KEY | — |
Required IAM Roles
| Operation | Roles required |
|---|---|
| Read | roles/bigquery.dataViewer + roles/bigquery.jobUser |
| Write | roles/bigquery.dataEditor + roles/bigquery.jobUser |
Write via the Spark connector (bigquery_temp_gcs_bucket set) | The above, plus roles/storage.objectAdmin (or at minimum object create/delete) on the staging bucket |
Spark Write Behaviour
Spark writes use format: 'bigquery' via spark-bigquery-with-dependencies.jar. Writes set createDisposition = CREATE_IF_NEEDED, so the destination table is created automatically if it doesn't already exist.
Known Limitations
- Spark-based writes cannot skip the GCS staging bucket —
bigquery_temp_gcs_bucketis mandatory for that path, and the connection's identity needs write access to it in addition to BigQuery roles. - ETL-Native reads/writes go via the BigQuery REST + Storage Read APIs and do not require a staging bucket, but also don't benefit from the Spark connector's bulk-load characteristics.