MongoDB Setup

Prerequisites

MongoDB 4.0 or higher, as a replica set, sharded cluster, MongoDB Atlas, or Amazon DocumentDB (engine version 5.0 or higher).
Network connectivity from the Supermetal agent to the deployment (default port 27017). For Atlas, allow the agent in your network access rules.

Change streams requirement

Change streams require a replica set or sharded cluster. A standalone MongoDB server must be converted to a single node replica set. Amazon DocumentDB requires change streams to be explicitly enabled per database (see Setup).

Replication Modes

Supermetal offers two replication modes for MongoDB's flexible document model.

Schema mode. Infers a typed schema from your documents, each field becoming a typed column in the target. New fields merge in as they appear. Best for analytics.
Schemaless mode. Preserves documents as JSON in two columns (_id, document). Supports parallel snapshots for faster initial loads. Best for variable documents or downstream JSON processing.

Consider this document.

{
  "_id": ObjectId("507f1f77bcf86cd799439011"),
  "name": "Alice",
  "email": "[email protected]",
  "age": 30,
  "is_active": true,
  "tags": ["admin", "user"]
}

Schema mode replicates a typed column per field.

`_id` (Utf8)	`name` (Utf8)	`email` (Utf8)	`age` (Int32)	`is_active` (Boolean)	`tags` (Utf8)
`507f1f77bcf86cd799439011`	`Alice`	`[email protected]`	`30`	`true`	`["admin","user"]`

Schemaless mode stores it in two columns.

`_id` (Utf8)	`document` (Json)
`507f1f77bcf86cd799439011`	`{"name":"Alice","email":"[email protected]","age":30,"is_active":true,"tags":["admin","user"]}`

Arrays

In schema mode, arrays are serialized as JSON strings to avoid schema conflicts when element types vary across documents.

Document Flattening

Nested documents can optionally flatten into top level columns, in both replication modes, using double underscore notation. The flattening depth can be capped, leaving subtrees below the cap as JSON. Arrays do not count toward depth.

// Original MongoDB document
{
  "_id": ObjectId("..."),
  "user": {
    "name": "John",
    "address": {
      "city": "San Francisco",
      "country": "USA"
    }
  },
  "tags": ["tag1", "tag2"]
}

// Flattened representation
{
  "_id": "...",
  "user__name": "John",
  "user__address__city": "San Francisco",
  "user__address__country": "USA",
  "tags": ["tag1", "tag2"]
}

Schema Mode

Typed String Inference

Schema mode can optionally infer numeric and temporal types from string values, so a field holding "123" lands as an integer column and "2024-01-01" as a date column. A majority vote across documents picks each field's type.

Staging

Schema mode stages the entire collection during the initial snapshot to infer types. Staging defaults to local storage, and large collections can exhaust local disk, so point the staging object store at S3, GCS, or Azure Blob instead.

Setup

Supermetal requires a dedicated MongoDB user with permissions to read data and change streams.

MongoDB Deployment	Minimum Required Permissions
Self managed	`read` role on the database to replicate from
MongoDB Atlas	`readAnyDatabase` role
Amazon DocumentDB	`read` role on the database to replicate from, plus `clusterMonitor` role on `admin`

Create a dedicated read only MongoDB user

Connect to your MongoDB instance using the mongo shell with admin privileges:

mongosh --host <host> --port <port> -u <admin-username> -p <admin-password> --authenticationDatabase admin

Script variables

Replace the placeholders with your own values:

<host>: your MongoDB server hostname or IP address
<port>: MongoDB port (default 27017)
<admin-username>: username with admin privileges
<admin-password>: password for the admin user

Create a dedicated user for Supermetal:

use admin
db.createUser({
  user: "supermetal_user",
  pwd: "strong-password",
  roles: [
    { role: "read", db: "target-database" }
  ]
})

Script variables

Replace strong-password with a unique password and target-database with the database you want to replicate from.

Log in to your MongoDB Atlas account

Visit cloud.mongodb.com to access your account.

Select your project and navigate to the Database Access section

Click the "Add New Database User" button

Enter a username and password

Under "Database User Privileges", select "Built-in Role" and choose "Read Any Database"

Optionally, restrict access to the clusters that contain the databases you want to replicate

Click "Add User" to create the user with the selected permissions

Create a dedicated user for Supermetal:

use admin
db.createUser({
  user: "supermetal_user",
  pwd: "strong-password",
  roles: [
    { role: "read", db: "target-database" },
    { role: "clusterMonitor", db: "admin" }
  ]
})

Script variables

Replace strong-password with a unique password and target-database with the database you want to replicate from.

Required roles

read: read data and change streams from the target database.
clusterMonitor: verify change stream configuration during connection validation.

Enable change streams on the target database:

Amazon DocumentDB requires change streams to be explicitly enabled per database. Run the following command as an admin user:

db.adminCommand({
  modifyChangeStreams: 1,
  database: "target-database",
  collection: "",
  enable: true
})

Change stream retention

Change stream events are retained for 3 hours by default (configurable up to 7 days via the change_stream_log_retention_duration cluster parameter). If the connector is paused longer than the retention window, a full re snapshot is required.

Network access

Amazon DocumentDB is VPC only and provides no public endpoints. The Supermetal agent must run within the same VPC as your DocumentDB cluster, or connect through a VPN or SSH tunnel.

IAM authentication

Amazon DocumentDB supports IAM authentication as an alternative to password auth. Add authMechanism=MONGODB-AWS to the connection string and credentials are sourced from the EC2 instance's IAM role.

On this page