BigQuery

Supermetal replicates to BigQuery through load jobs that bulk load from a Cloud Storage buffer.

Prerequisites

• A Google Cloud project with billing enabled and the BigQuery API activated.
• IAM access to create a service account, grant it project roles, and create datasets. This usually means roles/owner, or roles/iam.serviceAccountAdmin together with roles/resourcemanager.projectIamAdmin.
• A Cloud Storage bucket in the same region as the target dataset, since BigQuery load jobs read only from GCS.

Setup

gcloud iam service-accounts create supermetal \
  --project="my-project-123" \
  --display-name="Supermetal"

PROJECT="my-project-123"
SA="supermetal@${PROJECT}.iam.gserviceaccount.com"

# Run load/query jobs and create datasets
gcloud projects add-iam-policy-binding "$PROJECT" \
  --member="serviceAccount:${SA}" \
  --role="roles/bigquery.user"

# Read and write tables and data within datasets
gcloud projects add-iam-policy-binding "$PROJECT" \
  --member="serviceAccount:${SA}" \
  --role="roles/bigquery.dataEditor"

gcloud storage buckets add-iam-policy-binding "gs://my-buffer-bucket" \
  --member="serviceAccount:[email protected]" \
  --role="roles/storage.objectAdmin"

gcloud iam service-accounts keys create supermetal-key.json \
  --iam-account="[email protected]"

Multi table transactions

Opt in. When enabled, Supermetal wraps DML across multiple tables in a BigQuery script transaction. Changes from a single source transaction apply fully or not at all.

Query Priority

Supermetal submits query jobs (DDL, MERGE, scripts) at Interactive priority by default. Load jobs are unaffected.

Table Naming

Target table names carry the source schema as a prefix by default, so public.users becomes public_users. Turning the schema prefix off writes to users instead, which is safe only when table names are unique across all source schemas. Independently, you can add a custom prefix or suffix to every table name (raw_users, users_log).

Data Types Mapping

Changelog