File

FileSource ingests files from object stores (S3, GCS, Azure Blob) and filesystems. It handles format detection, compression, and incremental syncs automatically.

Features

Supported Backends

S3-compatible stores (MinIO, Cloudflare R2) use the S3 backend with a custom endpoint.

Supported Formats

Compression

Compression is detected from file extension. Example: data.csv.gz is detected as gzip-compressed CSV.

Archives

Prerequisites

Before you begin, ensure you have:

• Source Access: Read access to the object store, filesystem, or remote server containing your files
• Credentials:
  • AWS S3: IAM user or role with s3:GetObject and s3:ListBucket permissions
  • GCS: Service account with Storage Object Viewer role
  • Azure: SAS token with Read and List permissions
  • Local Filesystem: Read permissions on the directory
  • SFTP/FTP (coming soon): Username and password or SSH key
• Network Connectivity: Supermetal agent can reach the source endpoint
• (Optional) Write Access: Required only if using post-processing (delete or move files after sync)

Setup

aws iam create-policy \
    --policy-name supermetal-file-source-policy \
    --policy-document file://policy.json

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SupermetalFileSourcePolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket"
            ],
            "Resource": [
                "arn:aws:s3:::your-bucket/*",
                "arn:aws:s3:::your-bucket"
            ]
        }
    ]
}

end=$(date -u -d "1 year" '+%Y-%m-%dT%H:%MZ')

az storage container generate-sas \
    --name your-container \
    --account-name your-storage-account \
    --permissions rl \
    --expiry $end \
    --output tsv

file:///path/to/your/files

File Selection

Table Mapping

prefix: "raw_"

s3://exports/
├── customers.parquet
├── products.parquet
└── transactions.parquet

├── raw_customers
├── raw_products
└── raw_transactions

destination: "orders"

s3://vendor-data/
├── orders_jan.csv
├── orders_feb.csv
└── orders_mar.csv

└── orders

pattern: "(?P<entity>[^/]+)/(?P<year>[0-9]{4})/.*"
template: "{entity}_{year}"

s3://datalake/
├── sales/2024/q1.parquet
├── sales/2024/q2.parquet
├── orders/2024/q1.parquet
└── orders/2024/q2.parquet

├── sales_2024
└── orders_2024

Format Options

Polling

Set to 0 for a one-time sync.

Error Handling

Post-Processing

Action to take on source files after successful processing. Post-processing is best effort.

Move example: Files in s3://bucket/inbox/ moved to s3://bucket/processed/ after sync.

Limitations

• Nested archives not supported (e.g., .tar.gz containing .zip)
• Object store API rate limits apply (S3, GCS, Azure). Supermetal retries throttled requests automatically.