File

Supermetal writes snapshot and CDC rows to object stores (S3, GCS, Azure Blob) and filesystems in Parquet, CSV, JSON, or Avro format. It can optionally organize output files with Hive style partitioning.

Prerequisites

Create the S3 or GCS bucket, Azure container, Drive or Dropbox folder, SFTP root, or local directory that will hold the output files.
Allow network traffic from the Supermetal agent to the provider API or SFTP host and port.
Keep provider administrator access to create IAM principals, service accounts, OAuth apps, storage credentials, or SFTP users.

Setup

Configure AWS S3

Create an IAM Policy

Create a policy for the bucket and attach it to the IAM user or role that Supermetal uses.

Open the AWS IAM console.
Select Policies, then Create policy.
Select JSON and paste the policy shown below.
Select Next, name the policy, then select Create policy.

Save the policy below as policy.json, then create it with the AWS CLI.

aws iam create-policy \
  --policy-name supermetal-file-target-policy \
  --policy-document file://policy.json

Replace your-bucket with the bucket name.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::your-bucket"
    },
    {
      "Sid": "ManageOutputObjects",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:AbortMultipartUpload"
      ],
      "Resource": "arn:aws:s3:::your-bucket/*"
    }
  ]
}

In the IAM console, open the user or role, select Add permissions, then attach the new policy. With the AWS CLI, copy the policy ARN from the create command and run the command for the chosen principal. Replace the account ID and principal names.

aws iam attach-user-policy \
  --user-name supermetal-file-target \
  --policy-arn arn:aws:iam::123456789012:policy/supermetal-file-target-policy

aws iam attach-role-policy \
  --role-name supermetal-file-target \
  --policy-arn arn:aws:iam::123456789012:policy/supermetal-file-target-policy

Choose an Authentication Method

Attach the policy to the EC2 instance profile or ECS task role used by the agent. You can instead set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in the agent environment. Add AWS_SESSION_TOKEN for temporary environment credentials. Shared AWS profile files are ignored. Leave the access key values empty in Supermetal.

Open Users in the IAM console and select the user with the policy. Open Security credentials, select Create access key, then complete the access key flow. Keep the access key ID and secret access key.

Attach the policy to a role that the current AWS identity can assume. Request temporary credentials from AWS Security Token Service.

aws sts assume-role \
  --role-arn arn:aws:iam::123456789012:role/supermetal-file-target \
  --role-session-name supermetal-file-target

Keep the access key ID, secret access key, and session token from the response.

Open the IAM role and select Trust relationships, then Edit trust policy. Allow the identity provider to call sts:AssumeRoleWithWebIdentity. Set AWS_WEB_IDENTITY_TOKEN_FILE for the agent process. Keep the role ARN and an optional role session name. Leave the access key values empty.

For S3 compatible storage, use the full service endpoint. Keep Virtual Hosted Style Request on when the endpoint includes the bucket in its hostname. Turn it off when the service expects the bucket in the request path.

Connection Details

You need these values.

Keep the bucket name and AWS region.
Agent credentials need no credential value in Supermetal.
Access keys require an access key ID and secret access key.
Temporary credentials require an access key ID, secret access key, and session token.
Web identity requires a role ARN and can include a role session name.
S3 compatible storage requires an endpoint URL and the correct addressing style.

Configure Google Cloud Storage

Create a Service Account

Create a service account and grant it the Storage Object User role on the bucket.

Open IAM & Admin and select Service Accounts in the Google Cloud console.
Select Create service account, enter a name, then select Done.
Open Cloud Storage, select the bucket, then select Permissions.
Select Grant access, add the service account email, and assign Storage Object User.

Create the service account and grant access to the bucket.

gcloud iam service-accounts create supermetal-file-target \
  --project=your-project

gcloud storage buckets add-iam-policy-binding gs://your-bucket \
  --member=serviceAccount:[email protected] \
  --role=roles/storage.objectUser

Choose an Authentication Method

Configure Application Default Credentials that resolve to a service account. On Google Cloud, attach the service account to the compute resource that runs the agent.

Enable the IAM Service Account Credentials API. Grant the service account the Service Account Token Creator role on itself so it can sign requests.

gcloud services enable iamcredentials.googleapis.com \
  --project=your-project

gcloud iam service-accounts add-iam-policy-binding \
  [email protected] \
  --member=serviceAccount:[email protected] \
  --role=roles/iam.serviceAccountTokenCreator

Leave the service account key value empty.

Open the service account in the Google Cloud console. Select Keys, Add key, Create new key, then JSON. Store the downloaded file securely and use its complete contents.

You can also create the key with the Google Cloud CLI.

gcloud iam service-accounts keys create service-account.json \
  --iam-account=supermetal-file-target@your-project.iam.gserviceaccount.com

Connection Details

You need these values.

Keep the bucket name.
Leave the service account key value empty when using Application Default Credentials.
Keep the complete service account JSON when using JSON key authentication.

Configure Azure Blob Storage

Prepare the Container

Create a private container in the storage account.

Open the storage account in the Azure portal.
Select Containers, then Container.
Enter the container name, keep anonymous access disabled, then select Create.

Create the container with Microsoft Entra authentication.

az storage container create \
  --account-name your-storage-account \
  --name your-container \
  --auth-mode login

Choose an Authentication Method

Open the storage account in the Azure portal.
Select Containers, open the destination container, then select Shared access tokens.
Select Read, Write, Delete, and List permissions. Set the start and expiry times.
Select Generate SAS token and URL, then copy the SAS token.

Generate a container SAS token with Read, Write, Delete, and List permissions.

Sign in with an identity that can read the storage account keys. The commands create a SAS token that expires after one year.

account_key=$(az storage account keys list \
  --resource-group your-resource-group \
  --account-name your-storage-account \
  --query '[0].value' \
  --output tsv)

expiry=$(python3 -c 'from datetime import datetime, timedelta, timezone; print((datetime.now(timezone.utc) + timedelta(days=365)).strftime("%Y-%m-%dT%H:%MZ"))')

az storage container generate-sas \
  --account-name your-storage-account \
  --name your-container \
  --permissions rwdl \
  --account-key "$account_key" \
  --expiry "$expiry" \
  --output tsv

Open the storage account in the Azure portal. Select Access keys, then copy either key.

You can also read an access key with the Azure CLI.

az storage account keys list \
  --resource-group your-resource-group \
  --account-name your-storage-account \
  --query '[0].value' \
  --output tsv

Connection Details

You need these values.

Keep the storage account name.
Keep the container name.
Keep either the SAS token or the storage account access key.

Configure Local Filesystem

Create an absolute directory on the machine or container that runs the Supermetal agent.

Give the agent operating system account read, write, and execute permissions on the directory. These permissions must allow directory creation and removal plus file creation, listing, reading, renaming, and deletion.

You need the absolute directory path.

Configure Google Drive

Create an OAuth Client

Open the Google Drive API in the Google Cloud console, choose the project, then select Enable.
Open Google Auth Platform, then complete Branding and Audience for the Google account that owns the destination.
Open Data Access, select Add or remove scopes, then add https://www.googleapis.com/auth/drive.
Open Clients, select Create client, then create the OAuth client used by your authorization flow.

Choose an Authentication Method

Complete the Google OAuth server flow with the Drive scope. Exchange the authorization code for an access token and keep that token. Repeat the flow when the token expires.

Run the Google OAuth server flow with access_type=offline and prompt=consent. Exchange the authorization code and keep the OAuth client ID, client secret, and refresh token. Supermetal uses them to renew access.

Connection Details

You need these values.

Access token authentication requires a current access token.
Refresh token authentication requires the OAuth client ID, client secret, and refresh token.
The root folder is optional and starts below the Drive root.

Configure Dropbox

Create a Dropbox App

Open the Dropbox App Console and select Create app.
Choose App Folder access for an isolated app folder or Full Dropbox access for another folder in the account.
Open Permissions and enable files.content.read, files.content.write, and files.metadata.read.
Authorize the app for the Dropbox account that owns the destination.

Choose an Authentication Method

Open Settings in the Dropbox App Console. Under OAuth 2, select Generate for the generated access token. Keep the token and replace it when it expires.

Open this authorization URL after replacing APP_KEY.

https://www.dropbox.com/oauth2/authorize?client_id=APP_KEY&response_type=code&token_access_type=offline

Authorize the app and copy the authorization code. Exchange it for a refresh token with the app key and app secret.

curl https://api.dropboxapi.com/oauth2/token \
  --user 'APP_KEY:APP_SECRET' \
  --data code=AUTHORIZATION_CODE \
  --data grant_type=authorization_code

Keep the refresh token, app key, and app secret. The app key is the OAuth client ID. The app secret is the OAuth client secret. The Dropbox OAuth guide explains the complete flow.

Connection Details

You need these values.

Access token authentication requires a current access token.
Refresh token authentication requires the app key, app secret, and refresh token.
The root folder is optional and starts below the authorized Dropbox namespace.

Configure SFTP

Create an SFTP Account

Create a dedicated user with the SFTP server account manager. Create the output directory and make that user its owner. On Linux, create the directory after the user exists.

sudo install -d -m 0700 \
  -o supermetal -g supermetal \
  /srv/supermetal-files

Grant the user permission to create directories and to list, create, read, write, rename, and delete files below the remote root.

Create an OpenSSH Key

Generate an unencrypted OpenSSH key pair on a secure machine.

ssh-keygen -t ed25519 -N '' -f supermetal-file-target

Add supermetal-file-target.pub to the SFTP user's authorized_keys file. Keep the private key for Supermetal.

Pin the Server Host Key

Ask the server administrator for the OpenSSH host public key and verify its fingerprint through a trusted channel. Keep the full public key so Supermetal can reject a different server key.

Connection Details

You need these values.

Keep the host and port. The default port is 22.
Keep the SFTP user and unencrypted OpenSSH private key.
Keep the OpenSSH server public key.
The remote root is optional.

Output Format

Parquet is the default output format. Every format uses Zstandard compression by default.

Option	Behavior
Version	Parquet `1.0` is the default. Parquet `2.0` enables newer encodings.
Compression	`None`, `Gzip`, or `Zstandard`.
Compression level	`0` uses the codec default. Gzip accepts `1` through `9`. Zstandard accepts `1` through `22`. Disabled compression requires `0`.
Row group size	The default maximum is `1,048,576` rows.
Data page size	The default target is `1,048,576` uncompressed bytes.
Dictionary encoding	Enabled by default.
Statistics	`Page` is the default. `Chunk` writes row group statistics. `None` disables statistics. Nested schemas always disable statistics.

Option	Behavior
Compression	`None`, `Gzip`, or `Zstandard`.
Header	Column names are written by default.
Delimiter	A single byte. The default is a comma.
Quote	A single byte. The default is a double quote.
Quote escaping	Quotes are doubled by default. When doubling is disabled, the escape character defaults to a backslash.
Line terminator	`LF`, `CRLF`, or one byte. The default is `LF`.
Dialect rules	Delimiter and quote must differ. They cannot be `CR` or `LF`. A custom line terminator must differ from both. When quote doubling is disabled, escape must contain one byte and differ from delimiter, quote, and the line terminator. Escape cannot be `CR` or `LF`.
Null value	An empty field by default, or a custom value.
Encoding	UTF-8 by default. Output fails when the selected encoding cannot represent a value.

CSV provides separate format patterns for date, date and time, time, timestamp, and timestamp with time zone values. Empty patterns use these defaults.

Value	Default Text
Date	`2026-03-16`
Date and time	`2026-03-16T11:33:20.123`
Time	`11:33:20.123`
Timestamp	`2026-03-16T11:33:20.123456789`
Timestamp with time zone	UTC values such as `2026-03-16T11:33:20.123456789Z`.

Option	Behavior
Compression	`None`, `Gzip`, or `Zstandard`.
Layout	`NDJSON` writes one object per line and is the default. `Array` writes one array per file.
Null fields	`Include` writes explicit null values and is the default. `Omit` leaves null fields out.
Flattening	Disabled by default. When enabled, nested structs become top level fields. Arrays and maps stay nested.
Flatten separator	Double underscores are the default. A custom separator must produce unique field names.

Avro writes Object Container Files with None, Deflate, Zstandard, Bzip2, or XZ compression.

Type Group	Behavior
Integers	Signed integers through `Int64` and unsigned integers through `UInt32`. `UInt64` is rejected.
Floating point	`Float16`, `Float32`, and `Float64`.
Strings and binary	Arrow string and binary types, plus positive fixed size binary values.
Dates and times	`Date32`, millisecond `Time32`, microsecond `Time64`, and millisecond or microsecond timestamps. `Date64`, second or nanosecond timestamps, and other time units are rejected.
Decimals	`Decimal128` and `Decimal256` with a nonnegative scale no greater than the precision.
Nested values	Lists, structs, unions, and maps with nonnull string keys.
Dictionaries	`Int32` keyed `Utf8` dictionaries with Avro enum metadata. Other dictionaries are rejected.
Names	Field names must be valid Avro names and unique within their record.

Output Path

Supermetal uses this path by default.

{connector_id}/{database}/{schema}/{table}/dt={date}

The path keeps each connector, database, schema, table, and UTC date in a separate output directory. Removing {connector_id} lets multiple connectors share the same output namespace.

The path template accepts these variables.

Variable	Value
`{connector_id}`	Connector identifier.
`{database}`	Target database name, or `default` when absent.
`{schema}`	Target schema name, or `default` when absent.
`{table}`	Target table name.
`{year}`	Current UTC year with four digits.
`{month}`	Current UTC month with two digits.
`{day}`	Current UTC day with two digits.
`{date}`	Current UTC date as `YYYY-MM-DD`.

Date variables use UTC when Supermetal creates each file.

Hive Style Partitioning

Partitioning adds ordered key=value directories below the table output path. For example, an unpartitioned table can write below this directory.

orders/dt=2026-08-06/

An Identity field for region followed by a Year field for created_at changes the directory to this path.

orders/dt=2026-08-06/region=west/created_at_year=2026/

Field order controls directory nesting. Moving created_at before region places the year directory first.

Transform	Directory Value
Identity	The complete source value.
Year	The four digit UTC calendar year.
Month	The UTC calendar month as `YYYY-MM`.
Day	The UTC calendar day as `YYYY-MM-DD`.
Hour	The UTC calendar hour as `YYYY-MM-DD-HH`.

Identity uses the final output column name as its folder key. Calendar transforms add _year, _month, _day, or _hour. Override the folder key to match an existing partition naming convention.

Null partition values use the __HIVE_DEFAULT_PARTITION__ directory value.

An Identity partition removes its source column from each file when its folder key matches the final output column name and no Year, Month, Day, or Hour partition uses that column. To keep the column, set a different folder key for every Identity partition that uses it.

Target File Size

The target file size defaults to 256 MiB, with a minimum of 1 MiB.

Supermetal estimates size from uncompressed input, so compression does not change the target. A file can exceed the target when a batch or completed CDC transaction crosses it.

Prerequisites

Setup

Configure AWS S3

Create an IAM Policy

Choose an Authentication Method

Connection Details

Configure Google Cloud Storage

Create a Service Account

Choose an Authentication Method

Connection Details

Configure Azure Blob Storage

Prepare the Container

Choose an Authentication Method

Connection Details

Configure Local Filesystem

Configure Google Drive

Create an OAuth Client

Choose an Authentication Method

Connection Details

Configure Dropbox

Create a Dropbox App

Choose an Authentication Method

Connection Details

Configure SFTP

Create an SFTP Account

Create an OpenSSH Key

Pin the Server Host Key

Connection Details

Output Format

Output Path

Hive Style Partitioning

Target File Size

Changelog

On this page