Iceberg
Apache Iceberg is an open table format for large analytic datasets. Supermetal writes Parquet data files directly to Iceberg tables using REST, Glue, or S3 Tables catalogs with S3 or GCS storage.
Spec Versions
Iceberg V1 introduced schema evolution, hidden partitioning, and snapshot isolation. V2 added row-level deletes through position and equality delete files. V3 brings deletion vectors (replacing positional deletes), row-level lineage tracking, the variant type for semi-structured data, and geospatial types.
Supermetal defaults to V3 for new tables. Use V2 if your query engine doesn't support V3 yet.
Write Modes
Merge on Read (default)
SELECT * returns the current state of your data. When rows are updated or deleted, Supermetal writes equality delete files that mask older versions at query time. Duplicate primary keys are resolved using positional deletes (V2) or deletion vectors (V3). No data is rewritten at ingest.
Delete modes: Soft delete (default) preserves deleted rows, queryable via WHERE _sm_deleted = true. Hard delete removes rows completely from query results.
Requires V2 or V3 and a query engine that supports equality deletes (Spark 3.x+, Trino, Dremio, Snowflake, StarRocks).
Append
All changes are appended as new rows. Inserts, updates, and deletes each produce a new data row with metadata columns _sm_deleted and _sm_version. To query current state, filter with WHERE _sm_deleted = false and deduplicate by primary key using _sm_version.
Works with any Iceberg version and any query engine.
Comparison
| Merge on Read | Append | |
|---|---|---|
| Iceberg version | V2, V3 | V1, V2, V3 |
| Query engine support | Requires equality delete support | Any engine |
| Query complexity | SELECT * returns current state | Requires dedup logic |
| Read performance | Engine applies deletes at read time | Engine scans all versions |
Compaction
File creation rate is controlled by flush interval (default: 10 seconds). Run periodic compaction/maintenance using your query engine (Spark, Trino) or a table management service to optimize read performance.
Prerequisites
You need a catalog (REST, Glue, or S3 Tables), storage credentials for where data files will be written (S3 or GCS), and a target namespace for table creation.
Setup
Catalog
Configure the Iceberg catalog where table metadata is stored.
| Field | Description |
|---|---|
| URI | Catalog endpoint (e.g., https://catalog.example.com) |
| Warehouse | Storage location identifier |
| Authentication | OAuth2, Bearer, Basic, or SigV4 |
Authentication methods:
| Method | Use Case |
|---|---|
| OAuth2 | Production environments with token endpoint, client ID/secret |
| Bearer | Service accounts, CI/CD with static token |
| Basic | Development, JDBC catalogs with username/password |
| SigV4 | AWS services requiring request signing (region, service) |
| Field | Description |
|---|---|
| Warehouse | S3 location (e.g., s3://my-bucket/warehouse) |
| Region | AWS region |
| Catalog ID | AWS account ID (optional) |
| Credentials | Access key and secret |
| Field | Description |
|---|---|
| Table Bucket ARN | S3 Tables bucket ARN |
| Region | AWS region |
| Credentials | Access key and secret |
Target Namespace
Tables are created under this namespace. For multi-level namespaces, use comma-separated values: my_database, my_schema creates tables under my_database.my_schema.
Storage Credentials
Credentials for writing Parquet data files to cloud storage.
| Field | Description |
|---|---|
| Access Key ID | AWS access key |
| Secret Access Key | AWS secret key |
| Region | AWS region (e.g., us-east-1) |
| Endpoint | Custom endpoint for S3-compatible storage |
| Path Style Access | Enable for MinIO and similar |
| Field | Description |
|---|---|
| Credentials JSON | Service account key (base64-encoded) |
| Project ID | GCP project identifier |
Write Options
Control how data is written to Iceberg tables. See Write Modes for details on Merge on Read vs Append.
| Field | Default | Description |
|---|---|---|
| Spec Version | V3 | Iceberg table format version |
| Write Mode | Merge on Read | How updates and deletes are handled |
| Delete Mode | Soft | For Merge on Read: Soft preserves audit trail, Hard removes rows |
| Truncate Table if exists | Off | Remove existing data before snapshot sync |
| Metadata Compression | Gzip | Compression for Iceberg metadata files |
| Flush Interval | 10000 ms | Commit frequency |
Parquet Settings
Configure the Parquet file format. Defaults work well for most workloads.
| Field | Default | Description |
|---|---|---|
| Compression | Zstd | Zstd, Snappy, Gzip, Lz4Raw, Brotli, or Uncompressed |
| Compression Level | 3 | Zstd (1-22), Gzip (0-9), or Brotli (0-11) |
| Target File Size | 512 MB | Files roll when exceeding this size |
| Parquet Version | V1 | V1 for compatibility, V2 for better encoding |
Variant Type (V3)
Semi-structured source types such as Postgres JSONB, MySQL JSON, and MongoDB documents are automatically mapped to the Iceberg variant type on V3 tables. Variant encodes nested JSON natively in Parquet's binary variant format, giving query engines columnar access to individual fields without JSON parsing.
Truncate Table if exists
This option is off by default. Enable it to atomically remove all existing data before the initial snapshot sync, preventing duplicate rows when recreating a connector.
The previous data remains accessible via Iceberg time travel, so you can roll back if the sync fails.
Snapshot Metadata
Each commit writes properties to the Iceberg snapshot summary for debugging and audit:
sm.connector_id,sm.run_id- identify which sync produced the snapshotsm.source.commit_ts- source commit timestamp (CDC only)sm.truncated_from_snapshot- previous snapshot ID (truncate only)
Query via SELECT * FROM table$snapshots.
Limitations
- Schema evolution: Data types promotion is not yet supported.
- Partitioning: Table Partitioning is not yet supported.
Data Types
Source types are converted to Iceberg-compatible types. Types without native Iceberg support are stored as strings.
| Arrow Type | Iceberg Type | Notes |
|---|---|---|
Boolean | boolean | |
Int8, Int16, Int32 | int | Widened to 32-bit |
UInt8, UInt16 | int | Widened to 32-bit |
Int64 | long | |
UInt32 | long | Widened to 64-bit |
UInt64 | decimal(20,0) | Exceeds long range |
Float16, Float32 | float | |
Float64 | double | |
Decimal128(p,s) | decimal(p,s) | |
Decimal256(p,s) | string | Exceeds decimal128 range |
Date32, Date64 | date | |
Time32, Time64 | time | Converted to microseconds |
Timestamp(s/ms/us, tz) | timestamptz | Converted to microseconds, UTC |
Timestamp(s/ms/us, None) | timestamp | Converted to microseconds |
Timestamp(ns, *) | long | Nanoseconds not supported |
Utf8, LargeUtf8, Utf8View | string | |
Binary, LargeBinary, BinaryView | binary | |
FixedSizeBinary(n) | fixed(n) | |
List<T>, LargeList<T> | list<T> | |
Map<K,V> | map<K,V> | |
Struct | struct | |
Duration, Interval, Union, Null | string |
| Arrow Type | Iceberg Type | Notes |
|---|---|---|
Boolean | boolean | |
Int8, Int16, Int32 | int | Widened to 32-bit |
UInt8, UInt16 | int | Widened to 32-bit |
Int64 | long | |
UInt32 | long | Widened to 64-bit |
UInt64 | decimal(20,0) | Exceeds long range |
Float16, Float32 | float | |
Float64 | double | |
Decimal128(p,s) | decimal(p,s) | |
Decimal256(p,s) | string | Exceeds decimal128 range |
Date32, Date64 | date | |
Time32, Time64 | time | Converted to microseconds |
Timestamp(s/ms/us, tz) | timestamptz | Converted to microseconds, UTC |
Timestamp(s/ms/us, None) | timestamp | Converted to microseconds |
Timestamp(ns, *) | long | Query engines lack nanosecond support |
Utf8, LargeUtf8, Utf8View | string | |
Binary, LargeBinary, BinaryView | binary | |
FixedSizeBinary(n) | fixed(n) | |
List<T>, LargeList<T> | list<T> | |
Map<K,V> | map<K,V> | |
Struct | struct | |
Utf8 with arrow.json extension | variant | |
Duration, Interval, Union, Null | string |
Apache Iceberg is a trademark of the Apache Software Foundation. No endorsement by the Apache Software Foundation is implied by the use of this mark.
Last updated on