MongoDB

Supermetal's MongoDB integration leverages MongoDB's native change streams capability to provide efficient, reliable data extraction with minimal impact on your database. This integration enables both initial snapshot capture and continuous replication of changes.

This guide covers the features, prerequisites, and configuration steps required to connect MongoDB with Supermetal.

Replication Modes

Supermetal offers two replication modes to handle MongoDB's flexible document model:

Schema Mode: Automatically infers and evolves a strongly-typed schema from your documents. Each field becomes a typed column in the target, enabling native SQL queries and type-safe analytics. Best for structured data and analytics workloads.

Schemaless Mode: Preserves documents as JSON in a two-column format (_id, document). Supports parallelized snapshots for faster initial loads. Ideal for highly variable document structures or when you need to preserve the original document format for downstream JSON processing.

Click each tab to see how a MongoDB document is replicated in each mode.

{
  "_id": ObjectId("507f1f77bcf86cd799439011"),
  "name": "Alice",
  "email": "[email protected]",
  "age": 30,
  "is_active": true,
  "tags": ["admin", "user"]
}

Features

Prerequisites

Before you begin, ensure you have:

• MongoDB Requirements:

  • Version: MongoDB 4.0 or higher (required for change streams support)
  • Supported Deployments:
    • MongoDB Community/Enterprise Server (replica set configuration)
    • MongoDB Atlas
  Change Streams Requirement

  Change streams require a replica set deployment. If you're using a standalone server, you must convert it to a single-node replica set to use change streams.

• Deployment Requirements:

  • Replica Set: Your MongoDB deployment must be configured as a replica set
  • Read Concern Majority: Supermetal uses read concern "majority" to ensure consistent reads
  • Database Permissions: User with appropriate permissions (see Setup)
  • Network Connectivity: Ensure Supermetal can reach your MongoDB deployment (default port: 27017)
  • TLS/SSL Support: Supermetal supports both unencrypted and TLS/SSL encrypted connections

• MongoDB Atlas Requirements:

  • Network Access: Configure network access rules to allow Supermetal to connect
  • Connection String: Use the connection string format that includes all replica set members

Setup

Permissions Overview

Supermetal requires a dedicated MongoDB user with appropriate permissions to read data and access change streams. The recommended approach is to create a dedicated read-only user.

Create a Dedicated Read-Only MongoDB User

Self-ManagedMongoDB Atlas

Connect to your MongoDB instance using the mongo shell with admin privileges:

mongosh --host <host> --port <port> -u <admin-username> -p <admin-password> --authenticationDatabase admin

Script Variables

Replace the placeholder values in the command with your actual information:

• <host>: Your MongoDB server hostname or IP address
• <port>: MongoDB port (default is 27017)
• <admin-username>: Username with admin privileges
• <admin-password>: Password for the admin user

Create a dedicated user for Supermetal:

use admin
db.createUser({
  user: "supermetal_user",
  pwd: "strong-password",
  roles: [
    { role: "read", db: "target-database" }
  ]
})

Script Variables

Replace the placeholder values in the script with your actual information:

• strong-password: Replace with a secure, unique password for the supermetal_user.
• target-database: The name of the database you want to replicate from.

Log in to your MongoDB Atlas account

Visit cloud.mongodb.com to access your account.

Select your project and navigate to the Database Access section

Click the "Add New Database User" button

Enter a username and password

Under "Database User Privileges", select "Built-in Role" and choose "Read Any Database"

Optionally, you can restrict access to specific clusters that contain the databases you want to replicate

Click "Add User" to create the user with the selected permissions

Configure MongoDB Source in Supermetal

You are now ready to configure MongoDB as a source within Supermetal. When configuring the source, select your preferred replication mode:

• Schema Mode: Infers strongly-typed schemas from documents, creating typed columns in the target. Best for analytics workloads.
• Schemaless Mode: Preserves documents as JSON in a two-column format (_id, document). Best for variable document structures.

Data Types Mapping