Google Cloud Storage

Supermetal can stage your data in a Google Cloud Storage bucket before writing it into your target database.

Prerequisites

A Google Cloud account with permissions to create and manage Cloud Storage buckets, service accounts, and IAM policies.

Setup

Create a Cloud Storage Bucket

Go to Cloud Storage and select Buckets.
Click Create.
Enter a globally unique bucket name (e.g., supermetal-gcs-buffer).
Select the region where your Supermetal agent runs.
Click Create.

gcloud storage buckets create gs://supermetal-gcs-buffer \
    --project=<project-id> \
    --location=<location>

Script variables

Replace these values.

<project-id> with your Google Cloud project ID.
<location> with the region where your Supermetal agent runs.
supermetal-gcs-buffer with a globally unique bucket name.

Bucket region

For a BigQuery target, the bucket must be in the same location as the target dataset. For other targets, place the bucket in the same region as your Supermetal agent to avoid network costs and latency between regions.

Create a Service Account

Go to IAM & Admin and select Service Accounts.
Click Create service account.
Enter a name (e.g., supermetal-gcs-buffer).
Click Create and continue, then click Done. Bucket access is granted in the next step.

gcloud iam service-accounts create supermetal-gcs-buffer \
    --display-name="Supermetal GCS buffer" \
    --project=<project-id>

Script variables

Replace <project-id> with your Google Cloud project ID.

Grant Access to the Bucket

Grant the service account the Storage Object User role on the buffer bucket. This role allows Supermetal to create, read, list, and delete buffer objects and multipart uploads.

Go to Cloud Storage and select Buckets.
Open the bucket and select the Permissions tab.
Click Grant access.
Enter the service account email as the new principal.
Select the Cloud Storage > Storage Object User role.
Click Save.

gcloud storage buckets add-iam-policy-binding gs://supermetal-gcs-buffer \
    --member="serviceAccount:supermetal-gcs-buffer@<project-id>.iam.gserviceaccount.com" \
    --role="roles/storage.objectUser"

Script variables

Replace <project-id> with your Google Cloud project ID and supermetal-gcs-buffer with your bucket name.

Configure Authentication

Choose one of the following authentication methods. A future release will add a third tab for HMAC key authentication through the GCS S3 compatible API.

Attach the service account to the compute resource that runs the Supermetal agent. Leave Service Account Key empty so Supermetal uses the resource's Application Default Credentials and keeps permanent key material out of Supermetal.

Application Default Credentials require permission to sign GCS URLs. Enable the IAM Service Account Credentials API and allow the attached service account to sign blobs with its own identity.

Go to APIs & Services and select Library.
Search for IAM Service Account Credentials API and click Enable.
Go to IAM & Admin and select Service Accounts.
Open the service account and grant its own email the Service Account Token Creator role on this service account.

gcloud services enable iamcredentials.googleapis.com \
    --project=<project-id>

gcloud iam service-accounts add-iam-policy-binding \
    supermetal-gcs-buffer@<project-id>.iam.gserviceaccount.com \
    --member="serviceAccount:supermetal-gcs-buffer@<project-id>.iam.gserviceaccount.com" \
    --role="roles/iam.serviceAccountTokenCreator" \
    --project=<project-id>

Script variables

Replace <project-id> with your Google Cloud project ID.

Use a JSON key when Application Default Credentials are not available.

Go to IAM & Admin and select Service Accounts.
Select the service account.
Open the Keys tab.
Click Add key, then select Create new key.
Select JSON and click Create.
Store the downloaded key securely. Paste its full JSON content into the Service Account Key field in Supermetal.

gcloud iam service-accounts keys create supermetal-gcs-key.json \
    --iam-account=supermetal-gcs-buffer@<project-id>.iam.gserviceaccount.com

Script variables

Replace <project-id> with your Google Cloud project ID. Paste the full contents of supermetal-gcs-key.json into the Service Account Key field in Supermetal.

Service account key

Service account keys do not expire automatically. Store the key securely and rotate it regularly.

Connection details

Configure the buffer with a bucket name and either Application Default Credentials or the full service account JSON key.

Prerequisites

Setup

Create a Cloud Storage Bucket

Create a Service Account

Grant Access to the Bucket

Configure Authentication

On this page