Metrics
Connector Metrics
These metrics track data flow and performance through the ETL pipeline from source to target systems.
| Category | Metric Name | Description | Unit | Type |
|---|---|---|---|---|
| Data | rows.in | Number of rows read | 1 | Counter (u64) |
rows.out | Number of rows written | 1 | Counter (u64) | |
bytes.in | Number of bytes read | By | Counter (u64) | |
bytes.out | Number of bytes written | By | Counter (u64) | |
| Latency | latency.total | Latency from source event timestamp to event being processed & written to target. End-to-end latency | ns | Histogram (u64) |
latency.in | Latency from source event timestamp to when the event was received | ns | Histogram (u64) | |
latency.out | Latency from when the event was received to when event written to target | ns | Histogram (u64) | |
latency.in.process | Latency to process source event | ns | Histogram (u64) | |
latency.out.process | Latency to process event before writing to target | ns | Histogram (u64) | |
| Buffer | ||||
| ↳ Data | buffer.rows.in | Number of rows read from the buffer | 1 | Counter (u64) |
buffer.rows.out | Number of rows written to the buffer | 1 | Counter (u64) | |
buffer.bytes.in | Number of bytes read from the buffer | By | Counter (u64) | |
buffer.bytes.out | Number of bytes written to the buffer | By | Counter (u64) | |
| ↳ Latency | buffer.latency.total | Latency from source event timestamp to event being processed & written to buffer | ns | Histogram (u64) |
buffer.latency.in | Latency from source event timestamp to when the event was received by buffer | ns | Histogram (u64) | |
buffer.latency.out | Latency from when the event was received to when event written to buffer | ns | Histogram (u64) | |
buffer.latency.in.process | Latency to process the event was received by buffer | ns | Histogram (u64) | |
buffer.latency.out.process | Latency to process event before writing to buffer | ns | Histogram (u64) | |
| ↳ Object Store | buffer.object_store.files.force-flush | Number of times buffer forced a flush (on no writes) | 1 | Counter (u64) |
buffer.object_store.files.abort | Failed file operations due to errors | 1 | Counter (u64) | |
buffer.object_store.files.create | Successfully created new file in object storage | 1 | Counter (u64) | |
buffer.object_store.files.delete | Files removed from object storage | 1 | Counter (u64) | |
buffer.object_store.files.get | Files retrieved from object storage | 1 | Counter (u64) | |
| Snapshot Progress | progress.snapshot | Current progress of initial snapshot load (0.0 to 100.0) | % | Gauge (f64) |
buffer.progress.snapshot | Current progress of buffer snapshot operations (0.0 to 100.0) | % | Gauge (f64) |
Process Metrics
System-level metrics for the Supermetal process following OpenTelemetry semantic conventions. These help monitor resource usage and system health.
| Category | Metric Name | Description | Unit | Type |
|---|---|---|---|---|
| Process CPU | process.cpu.time | Total CPU time consumed by the process since startup | s | Counter (u64) |
process.cpu.utilization | Current CPU usage as ratio (0.0=idle, 1.0=fully utilized) | 1 | Gauge (f64) | |
| Process Memory | process.memory.usage | Physical memory (RSS) currently used by the process | By | UpDownCounter (i64) |
process.memory.virtual | Virtual memory currently allocated by the process | By | UpDownCounter (i64) | |
| Process Resources | process.open_file_descriptor_count | Currently open file descriptors (monitor for fd leaks) | 1 | UpDownCounter (i64) |
process.thread_count | Currently active threads (monitor for thread leaks) | 1 | UpDownCounter (i64) | |
process.disk.io | Total disk I/O bytes with direction label (read/write) | By | Counter (u64) |
Runtime Metrics
Tokio async runtime metrics providing insights into task scheduling, worker thread utilization, and async I/O performance.
| Category | Metric Name | Description | Unit | Type |
|---|---|---|---|---|
| Runtime Tasks | runtime.global_queue_depth | Tasks waiting in global queue (high values indicate backpressure) | 1 | Gauge (u64) |
runtime.alive_tasks | Currently executing async tasks | 1 | Gauge (u64) | |
runtime.spawned_tasks | Total async tasks created since startup | 1 | Counter (u64) | |
runtime.budget_forced_yield_count | Tasks forced to yield due to execution time limits (indicates CPU-bound tasks) | 1 | Counter (u64) | |
| Runtime Workers | runtime.num_workers | Total worker threads in the runtime thread pool | 1 | Gauge (u64) |
runtime.blocking_queue_depth | Tasks waiting for blocking thread pool | 1 | Gauge (u64) | |
runtime.num_blocking_threads | Currently active blocking threads | 1 | Gauge (u64) | |
runtime.num_idle_blocking_threads | Idle blocking threads available for work | 1 | Gauge (u64) | |
runtime.io_driver_ready_count | I/O events processed by the runtime (network/file operations) | 1 | Counter (u64) | |
| Worker Metrics (opt-in) | runtime.worker_local_queue_depth | Tasks in each worker's local queue (set ENABLE_WORKER_METRICS=1) | 1 | Gauge (u64) |
runtime.worker_local_schedule_count | Tasks scheduled locally per worker (indicates work distribution) | 1 | Counter (u64) | |
runtime.worker_mean_poll_time | Average task execution time per worker | ns | Gauge (u64) | |
runtime.worker_noop_count | Idle polls per worker (high values indicate low workload) | 1 | Counter (u64) | |
runtime.worker_overflow_count | Local queue overflows per worker (tasks moved to global queue) | 1 | Counter (u64) | |
runtime.worker_park_count | Times worker thread went idle waiting for tasks | 1 | Counter (u64) | |
runtime.worker_poll_count | Total task polls per worker | 1 | Counter (u64) | |
runtime.worker_steal_count | Tasks stolen from other workers (indicates load balancing) | 1 | Counter (u64) | |
runtime.worker_steal_operations | Steal attempts per worker (successful + failed) | 1 | Counter (u64) | |
runtime.worker_total_busy_duration | Cumulative execution time per worker | ns | Counter (u64) |
Last updated on