Supermetal Documentation

Supermetal can stage your data in an AWS S3 bucket before writing it into your target database.

This guide walks you through configuring an AWS S3 bucket as a Supermetal buffer.

BYOC

AWS S3 Buffer is automatically managed by Supermetal Control Plane.

Prerequisites

Before you begin, ensure you have an AWS account with permissions to create and manage:

S3 buckets.
IAM roles and policies.

Setup

Create an S3 bucket

Go to the AWS S3 Console.
In the navigation bar on the top of the page, choose the name of the currently displayed AWS Region. Next, choose the Region in which you want to create a bucket.
In the left navigation pane, choose General purpose buckets.
Click on "Create bucket".
Enter a unique bucket name (supermetal-s3-buffer).
Configure optional settings as per your requirements.
Click on "Create bucket".

aws s3api create-bucket \
    --bucket supermetal-s3-buffer \
    --region <region> \
    --create-bucket-configuration LocationConstraint=<region>

Script Variables

Replace:

<region> with the region of the bucket you want to create.
supermetal-s3-buffer with the name of the bucket you want to create.

Bucket Region

The bucket region must match the region of your Supermetal agent to avoid cross region networking costs and latency.

Create IAM policy

Navigate to the AWS IAM Console.
Go to policies and click on "Create policy".
Select "JSON" and paste the following policy document:

Create a policy document file from the following template and run the following command:

aws iam create-policy \
    --policy-name <policy-name> \
    --policy-document file://<policy-document-file>

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SupermetalS3BufferPolicy",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload"
            ],
            "Resource": [
                "arn:aws:s3:::supermetal-s3-buffer/*",
                "arn:aws:s3:::supermetal-s3-buffer"
            ],
            "Effect": "Allow"
        }
    ]
}

Policy Variables

Replace supermetal-s3-buffer with the name of the bucket you want to create.

Create IAM user

Navigate to the AWS IAM Console.
Go to users and click on "Create user".
Enter a name for the user (e.g., "supermetal-s3-user").
Select Attach policies directly and attach the policy you created in the previous step.
Click "Create user".
Select the user and click on "Security credentials".
Click on "Create access key".
Select "CLI / Programmatic Access" and click on "Create access key".

aws iam create-user \
    --user-name supermetal-s3-user \
    --permissions-boundary <policy-arn>
aws iam create-access-key \
    --user-name supermetal-s3-user

Script Variables

Replace <policy-arn> with the ARN of the policy you created in the previous step (arn:aws:iam::<account-id>:policy/supermetal-s3-user-policy).

Connection Details

You'll need the following connection details to configure the buffer in Supermetal:

Access key
Secret access key
Bucket name
Region

AWS S3

Prerequisites

Setup

Create an S3 bucket

Create IAM policy

Create IAM user

On this page