Iceberg

Apache Iceberg is an open table format for large analytic datasets. Supermetal writes Parquet data files directly to Iceberg tables using REST, Glue, or S3 Tables catalogs with S3 or GCS storage.

---

Spec Versions

Iceberg V1 introduced schema evolution, hidden partitioning, and snapshot isolation. V2 added row-level deletes through position and equality delete files. V3 brings deletion vectors (replacing positional deletes), row-level lineage tracking, the variant type for semi-structured data, and geospatial types.

Supermetal defaults to V3 for new tables. Use V2 if your query engine doesn't support V3 yet.

---

Write Modes

Merge on Read (default)

SELECT * returns the current state of your data. When rows are updated or deleted, Supermetal writes equality delete files that mask older versions at query time. Duplicate primary keys are resolved using positional deletes (V2) or deletion vectors (V3). No data is rewritten at ingest.

Delete modes: Soft delete (default) preserves deleted rows, queryable via WHERE _sm_deleted = true. Hard delete removes rows completely from query results.

Requires V2 or V3 and a query engine that supports equality deletes (Spark 3.x+, Trino, Dremio, Snowflake, StarRocks).

Append

All changes are appended as new rows. Inserts, updates, and deletes each produce a new data row with metadata columns _sm_deleted and _sm_version. To query current state, filter with WHERE _sm_deleted = false and deduplicate by primary key using _sm_version.

Works with any Iceberg version and any query engine.

Comparison

---

Prerequisites

You need a catalog (REST, Glue, or S3 Tables), storage credentials for where data files will be written (S3 or GCS), and a target namespace for table creation.

---

Setup

---

Partitioning

Define partition specs per table to physically lay out data on disk by transform values (day, hash bucket, etc.). Query engines use these to prune files and accelerate scans.

Transforms

bucket(N) distributes rows across N hash buckets, useful for high-cardinality columns where you want even distribution. truncate(W) rounds integers, prefixes strings, or shortens binary to width W.

Iceberg rejects redundant transforms on the same column (for example both day(ts) and month(ts)).

Supermetal's metadata columns _sm_version (long) and _sm_deleted (boolean) can be used as partition sources.

---

Variant Type (V3)

Semi-structured source types such as Postgres JSONB, MySQL JSON, and MongoDB documents are automatically mapped to the Iceberg variant type on V3 tables. Variant encodes nested JSON natively in Parquet's binary variant format, giving query engines columnar access to individual fields without JSON parsing.

---

Truncate Table if exists

This option is off by default. Enable it to atomically remove all existing data before the initial snapshot sync, preventing duplicate rows when recreating a connector.

The previous data remains accessible via Iceberg time travel, so you can roll back if the sync fails.

---

Snapshot Metadata

Each commit writes properties to the Iceberg snapshot summary for debugging and audit:

• sm.connector_id, sm.run_id - identify which sync produced the snapshot
• sm.source.commit_ts - source commit timestamp (CDC only)
• sm.truncated_from_snapshot - previous snapshot ID (truncate only)

Query via SELECT * FROM table$snapshots.

---

Table Name Mapping

Supermetal preserves source table names on the target when possible, sanitizing only when the catalog rejects them as written.

REST compatible catalogs accept most characters and pass names through; whitespace is replaced with _ since storage paths can't contain it. Glue and S3 Tables require lowercase identifiers of letters, digits, and underscores, so Supermetal lowercases the name, replaces -, ., and whitespace with _, and drops other special characters.

When the first character isn't valid for the catalog (a leading digit on REST, a leading underscore on Glue or S3 Tables), Supermetal prepends t_ rather than stripping the character to avoid conflicts.

A source name with no characters valid for the catalog maps to t_ followed by the hex bytes of the original.

Reserved metadata names

Iceberg reserves identifiers like entries, files, history, manifests, partitions, snapshots, refs, and their all_ and _delete_files variants for metadata tables. A real table sharing one of these names collides with Iceberg's metadata table redirect and breaks reads on most catalogs. Supermetal suffixes such names with _tbl, so files becomes files_tbl. The full list is in MetadataTableType.

Examples

---

Limitations

• Schema evolution: Data type promotion is not yet supported.
• Partition spec evolution: Changing partitioning on an existing table requires recreating it.

---

Data Types

Source types are converted to Iceberg-compatible types. Types without native Iceberg support are stored as strings.

---

Apache Iceberg is a trademark of the Apache Software Foundation. No endorsement by the Apache Software Foundation is implied by the use of this mark.

Changelog