Skip to content

Metrics

Use metrics when you need counts, durations, throughput, storage failures, or small product-specific measurements from a Firecube run.

Firecube is a batch CLI, so it does not expose a /metrics endpoint. It buffers metrics while the command runs and pushes the final snapshot to Prometheus Pushgateway at the end when a Pushgateway is configured.

Enable Metrics

Set the Pushgateway URL before running Firecube:

export FIRECUBE_PUSHGATEWAY_URL="http://localhost:9091"

Or set it in ~/.config/firecube/config.toml:

[metrics]
pushgateway_url = "http://localhost:9091"

If no Pushgateway is configured, the command still runs. Metrics are just not pushed.

Disable metric pushing for an environment:

export FIRECUBE_METRICS_DISABLED=true

What Firecube Emits

The engine emits standard run metrics automatically: batch counts, file counts, pipeline timing, storage timing, validation timing, error counts, and related run-summary signals.

Plugin code should not emit those engine-owned metrics manually. Plugin metrics should be product-specific values that the engine cannot infer, such as records extracted, detections found, scenes parsed, or domain-specific quality counters.

Metric Kinds

Use counter for values that increase during a run, such as files parsed or records extracted. Prometheus names get _total when needed.

Use gauge for current values, sizes, durations, latest measurements, or values that can go up and down.

Use explicit units in metric names, for example _seconds, _bytes, or _rows.

Plugin Metrics

Plugin metrics are emitted through the injected ctx.telemetry object:

if ctx.telemetry is not None:
    ctx.telemetry.emit(
        "weather_csv_files",
        len(items),
        kind="counter",
        meta={"group": group, "status": "success"},
    )

See Plugin Observability for the full plugin contract, including ctx.telemetry.emit(...), span usage, and what plugins must not import.

Labels

Metric metadata becomes Prometheus labels only when the key is allowlisted. Good labels are bounded and low-cardinality:

meta={"group": "default", "status": "success"}

Avoid labels with unbounded values:

  • run_id
  • batch_id
  • timestamps
  • file paths or URIs
  • exception messages
  • stack traces

If operators need additional bounded labels, extend the allowlist in config or environment. See Observability Reference.

Standard Run Metrics

The complete standard metric table is generated from the canonical Firecube metric schema. The same table is also available in the Observability Reference.

Standard run metrics
Summary key Prometheus metric Kind Description
workers firecube_pipeline_workers gauge Pipeline worker count
batch_size firecube_pipeline_batch_size gauge Configured batch size
batches_total firecube_pipeline_batches_total counter Completed batch count
batches_failed firecube_pipeline_batches_failed_total counter Failed batch count
hook_failures firecube_pipeline_hook_failures_total counter Non-fatal lifecycle hook failures
files_processed firecube_files_processed_total counter Source files processed
bytes_ingested firecube_bytes_ingested_total counter Source bytes ingested
rows_processed firecube_rows_processed_total counter Rows processed when available
duration_total_s firecube_run_duration_seconds gauge End-to-end run duration
duration_pipeline_s firecube_pipeline_duration_seconds gauge Pipeline duration before staged upload
duration_processing_s firecube_pipeline_batch_duration_seconds gauge Batch processing duration
duration_batch_creation_s firecube_pipeline_batch_creation_duration_seconds gauge Batch creation duration
duration_upload_s firecube_pipeline_upload_duration_seconds gauge Staged upload duration
duration_cpu_s firecube_pipeline_cpu_duration_seconds gauge Estimated CPU time
non_cpu_wait_s firecube_pipeline_non_cpu_wait_seconds gauge Estimated non-CPU wait time
cpu_utilization_estimate firecube_pipeline_cpu_utilization_estimate gauge Estimated CPU utilization, bounded from 0 to 1
storage_client_requests firecube_storage_client_requests_total counter Storage client request count
storage_client_errors firecube_storage_client_errors_total counter Storage client errors
storage_client_retryable_errors firecube_storage_client_retryable_errors_total counter Retryable storage client errors
storage_client_latency_s_total firecube_storage_client_latency_seconds gauge Total observed storage client latency
storage_client_bytes_read firecube_storage_client_bytes_read_total counter Bytes read by storage client
storage_client_bytes_written firecube_storage_client_bytes_written_total counter Bytes written by storage client
wal_corruption_count firecube_control_plane_corruption_total counter Control-plane WAL corruption events
wal_torn_tail_recovery_count firecube_control_plane_torn_tail_recovery_total counter Torn WAL tails recovered automatically
wal_snapshot_rebuild_duration_s firecube_control_plane_snapshot_rebuild_duration_seconds gauge Snapshot rebuild duration
wal_snapshot_rebuild_count firecube_control_plane_snapshot_rebuild_total counter Snapshot rebuild count
resume_guard_enforce_duration_s firecube_resume_guard_enforce_duration_seconds gauge
resume_guard_runs_enumerated firecube_resume_guard_runs_enumerated_total counter
resume_guard_spans_scanned firecube_resume_guard_spans_scanned_total counter

Next Steps