File

Supermetal ingests files from object stores (S3, GCS, Azure Blob) and filesystems. Format, compression, and CSV dialect are detected from the files themselves. Syncs are incremental, processing only changed files. Every file streams through the pipeline without local disk or a file size cap.

Supported Backends

S3 compatible stores (MinIO, Cloudflare R2) use the S3 backend with a custom endpoint.

Supported Formats

Compression

Compression is detected from the file extension. data.csv.gz is detected as gzip compressed CSV.

Archives

Prerequisites

• Read access to the object store or filesystem holding your files. The Setup tabs cover credentials per backend.
• Network connectivity from the Supermetal agent to the source endpoint.
• Write access, only when post processing deletes or moves files after sync.

Setup

aws iam create-policy \
    --policy-name supermetal-file-source-policy \
    --policy-document file://policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SupermetalFileSourcePolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket/*",
                "arn:aws:s3:::your-bucket"
            ]
        }
    ]
}

end=$(date -u -d "1 year" '+%Y-%m-%dT%H:%MZ')

az storage container generate-sas \
    --name your-container \
    --account-name your-storage-account \
    --permissions rl \
    --expiry $end \
    --output tsv

file:///path/to/your/files

File Selection

Table Mapping

prefix: "raw_"

s3://exports/
├── customers.parquet
├── products.parquet
└── transactions.parquet

├── raw_customers
├── raw_products
└── raw_transactions

destination: "orders"

s3://vendor-data/
├── orders_jan.csv
├── orders_feb.csv
└── orders_mar.csv

└── orders

pattern: "(?P<entity>[^/]+)/(?P<year>[0-9]{4})/.*"
template: "{entity}_{year}"

s3://datalake/
├── sales/2024/q1.parquet
├── sales/2024/q2.parquet
├── orders/2024/q1.parquet
└── orders/2024/q2.parquet

├── sales_2024
└── orders_2024

Format Options

Polling

Supermetal discovers new and modified files by polling the source location, every 60 seconds by default. Setting the interval to 0 disables polling and runs the sync once.

Error Handling

When a file fails to process, the sync logs the error, skips the file, and continues (default), or stops on the first error.

Post Processing

After successful processing, source files can be left in place (default), deleted, or moved to another path, for example from s3://bucket/inbox/ to s3://bucket/processed/. Post processing is best effort (a failed delete or move does not fail the sync) and requires write access to the source bucket.

Limitations

• Nested archives not supported (e.g., .tar.gz containing .zip)
• Object store API rate limits apply (S3, GCS, Azure). Supermetal retries throttled requests automatically.

Changelog