MotherDuckMotherDuck

Supermetal replicates to MotherDuck through its Postgres wire protocol endpoint.

Prerequisites

  • A MotherDuck account with an access token.
  • An external object store configured as the Supermetal buffer. Local file paths are not supported over the Postgres endpoint.
  • A persistent secret in MotherDuck for the object store (see step 2 below).

Setup

Connect to MotherDuck

MotherDuck exposes a Postgres wire protocol endpoint. Use it to connect from Supermetal without running DuckDB locally.

FieldValue
Hostpg.<region>-aws.motherduck.com (e.g., pg.us-east-1-aws.motherduck.com)
Port5432
Userpostgres
PasswordYour MotherDuck access token
DatabaseDatabase name, or md: for the default database

Access token

See Authenticating to MotherDuck to create an access token.

Create object store secret

Create a secret so MotherDuck can read Parquet files from Supermetal's buffer (S3, Azure Blob Storage).

Open the MotherDuck SQL console and run one of the following.

CREATE SECRET IN MOTHERDUCK (
    TYPE S3,
    KEY_ID '...',
    SECRET '...',
    REGION 'us-east-1'
);
CREATE SECRET IN MOTHERDUCK (
    TYPE AZURE,
    CONNECTION_STRING 'AccountName=myaccount;AccountKey=...'
);

Target database and schema

FieldDefaultDescription
Target Database(required)Database to write into. Created automatically if it does not exist.
Target SchemaOptional. When set, all source tables land in this schema regardless of their source schema. Otherwise, each table keeps its source schema. Created if it does not exist.

Primary Keys

Opt in. Supermetal deduplicates using MERGE without primary key constraints on the table. Enabling primary keys adds the constraint, enforcing uniqueness at the database level.

Multi table transactions

Opt in. When enabled, each CDC batch wraps in a transaction, preserving source transaction boundaries across tables.

DuckLake

Supermetal can write to DuckLake databases on MotherDuck. Create a DuckLake database (fully managed or BYOB) and set it as the target database.

CREATE DATABASE my_ducklake (TYPE DUCKLAKE);

Data Types Mapping

Arrow TypeDuckDB TypeNotes
Int8TINYINT
Int16SMALLINT
Int32INTEGER
Int64BIGINT
UInt8UTINYINT
UInt16USMALLINT
UInt32UINTEGER
UInt64UBIGINT
Float16FLOAT
Float32FLOAT
Float64DOUBLE
Decimal128(p,s)DECIMAL(p,s)Falls back to VARCHAR when precision exceeds 38
Decimal256(p,s)DECIMAL(p,s)Falls back to VARCHAR when precision exceeds 38
Arrow TypeDuckDB TypeNotes
BooleanBOOLEAN
Arrow TypeDuckDB TypeNotes
Date32, Date64DATE
Time32, Time64TIME
Timestamp(s, None)TIMESTAMP_S
Timestamp(ms, None)TIMESTAMP_MS
Timestamp(us, None)TIMESTAMPMicrosecond precision, DuckDB's default
Timestamp(ns, None)TIMESTAMP_NS
Timestamp(ns, "UTC")TIMESTAMP_NS
Timestamp(*, "UTC")TIMESTAMPUTC timestamps except nanosecond
Timestamp(*, tz)TIMESTAMPTZTime zones other than UTC
IntervalINTERVAL
DurationVARCHAR
Arrow TypeDuckDB TypeNotes
Utf8, LargeUtf8VARCHAR
Arrow TypeDuckDB TypeNotes
Binary, LargeBinary, FixedSizeBinaryBLOB
Arrow TypeDuckDB TypeNotes
Utf8 with arrow.json extensionVARIANTDuckDB 1.5+. Falls back to JSON on older versions.
StructSTRUCT(...)Field names and types preserved
MapMAP(K, V)
Arrow TypeDuckDB TypeNotes
List<T>, LargeList<T>, FixedSizeList<T>T[]

Nullability

Nullable source columns are nullable on the target. All columns except primary keys are nullable by default to handle CDC edge cases. Enable preserve_source_nullability to carry over NOT NULL constraints from the source schema.

Changelog

Last updated on

On this page