Observability Reference

This reference lists Firecube's metrics, logs, traces, and Pushgateway configuration surface.

For the operational model, see Observability. For plugin telemetry rules, see Plugin Observability.

Metric Kinds

Kind	Meaning	Name normalization
`counter`	Value increments a running counter	Adds `_total` when missing
`gauge`	Value sets the current measurement	No suffix added

Metric names are sanitized for Prometheus and get a firecube_ prefix when missing.

Labels

Every pushed metric includes these base labels:

Label	Meaning
`plugin`	Plugin name
`product`	Product name
`output_format`	Output format such as `zarr` or `parquet`
`write_mode`	`direct` or `staged`

Default optional metadata labels:

group, forecast_horizon_hours, msg_region, satellite, status, error_type, resolution

Extend the label allowlist with either config or environment:

[metrics]
label_allowlist = ["frp_variant", "custom_label"]

export FIRECUBE_METRICS_LABEL_ALLOWLIST="frp_variant,custom_label"

Avoid labels with unbounded values such as run IDs, batch IDs, file paths, timestamps, exception messages, or stack traces.

Environment Variables

Variable	Default	Purpose
`FIRECUBE_LOG_FORMAT`	`json`	`json` or `plain`
`FIRECUBE_LOG_LEVEL`	`INFO`	Root logging level
`FIRECUBE_LOG_STRUCTURED_FIELDS`	default JSON field list	Comma-separated JSON log fields
`FIRECUBE_DEBUG`	unset	Enables DEBUG logs for Firecube namespaces
`OTEL_EXPORTER_OTLP_ENDPOINT`	unset	OTLP HTTP trace export endpoint
`OTEL_EXPORTER_OTLP_HEADERS`	unset	Headers passed to the OTLP exporter
`OTEL_DEBUG`	`false`	Emits spans to the console when no OTLP endpoint is set
`KFP_RUN_ID`	unset	Adds `kfp_run_id` to telemetry spans
`FIRECUBE_PUSHGATEWAY_URL`	unset	Pushgateway URL
`FIRECUBE_METRICS_DISABLED`	`false`	Disable metric pushing
`FIRECUBE_METRICS_LABEL_ALLOWLIST`	unset	Additional allowed metadata label keys
`FIRECUBE_METRICS_INCLUDE_RUN_ID_LABEL`	`false`	Add `run_id` as a metric label; not recommended
`FIRECUBE_PUSHGATEWAY_JOB`	`firecube`	Pushgateway job name
`FIRECUBE_PUSHGATEWAY_GROUPING_KEYS`	`instance,plugin,product`	Comma-separated grouping keys
`FIRECUBE_PUSHGATEWAY_GROUP_<KEY>`	unset	Value for custom grouping key `<key>`
`FIRECUBE_PUSHGATEWAY_DELETE_AFTER_PUSH`	`false`	Delete pushed metrics after a successful push

Trace Spans

Common ingest spans:

Span	Key attributes	When emitted
`firecube.cli.ingest`	`firecube.plugin`, `firecube.write_mode`	Ingest CLI command
`firecube.ingest`	`firecube.run_id`, `firecube.plugin`, `firecube.product`	Full plugin run
`firecube.batch`	`firecube.batch_id`	Each pipeline batch
`firecube.finalize`	none	Run finalization
`firecube.upload_s3`	`firecube.target`	Staged upload to S3

Template-specific spans:

Template path	Additional spans
`GenericZarrIngestor`	`firecube.batch.prepare`, `firecube.batch.zarr_write`
Tensogram output	`firecube.batch.prepare`, `firecube.batch.tensogram_write`
`GenericParquetIngestor`	Uses the common batch span unless the plugin emits its own spans
`DirectZarrIngestor`	Uses the common batch span unless the plugin emits its own spans

Config File

Firecube also reads metric settings from ~/.config/firecube/config.toml:

[metrics]
pushgateway_url = "http://localhost:9091"
label_allowlist = ["frp_variant", "custom_label"]

Environment variables override the config file for pushgateway_url. Label allowlists are additive: defaults, config values, and environment values are merged.

Pushgateway Grouping

By default, Firecube groups pushed metrics by:

instance,plugin,product

Use grouping keys for per-run separation instead of adding run_id as a metric label. For example:

export FIRECUBE_PUSHGATEWAY_GROUPING_KEYS="instance,plugin,product,workflow"
export FIRECUBE_PUSHGATEWAY_GROUP_WORKFLOW="nightly-fire"

Standard Run Metrics

These metrics are emitted from the run summary.

Summary key	Prometheus metric	Kind	Description
`workers`	`firecube_pipeline_workers`	gauge	Pipeline worker count
`batch_size`	`firecube_pipeline_batch_size`	gauge	Configured batch size
`batches_total`	`firecube_pipeline_batches_total`	counter	Completed batch count
`batches_failed`	`firecube_pipeline_batches_failed_total`	counter	Failed batch count
`hook_failures`	`firecube_pipeline_hook_failures_total`	counter	Non-fatal lifecycle hook failures
`files_processed`	`firecube_files_processed_total`	counter	Source files processed
`bytes_ingested`	`firecube_bytes_ingested_total`	counter	Source bytes ingested
`rows_processed`	`firecube_rows_processed_total`	counter	Rows processed when available
`duration_total_s`	`firecube_run_duration_seconds`	gauge	End-to-end run duration
`duration_pipeline_s`	`firecube_pipeline_duration_seconds`	gauge	Pipeline duration before staged upload
`duration_processing_s`	`firecube_pipeline_batch_duration_seconds`	gauge	Batch processing duration
`duration_batch_creation_s`	`firecube_pipeline_batch_creation_duration_seconds`	gauge	Batch creation duration
`duration_upload_s`	`firecube_pipeline_upload_duration_seconds`	gauge	Staged upload duration
`duration_cpu_s`	`firecube_pipeline_cpu_duration_seconds`	gauge	Estimated CPU time
`non_cpu_wait_s`	`firecube_pipeline_non_cpu_wait_seconds`	gauge	Estimated non-CPU wait time
`cpu_utilization_estimate`	`firecube_pipeline_cpu_utilization_estimate`	gauge	Estimated CPU utilization, bounded from 0 to 1
`storage_client_requests`	`firecube_storage_client_requests_total`	counter	Storage client request count
`storage_client_errors`	`firecube_storage_client_errors_total`	counter	Storage client errors
`storage_client_retryable_errors`	`firecube_storage_client_retryable_errors_total`	counter	Retryable storage client errors
`storage_client_latency_s_total`	`firecube_storage_client_latency_seconds`	gauge	Total observed storage client latency
`storage_client_bytes_read`	`firecube_storage_client_bytes_read_total`	counter	Bytes read by storage client
`storage_client_bytes_written`	`firecube_storage_client_bytes_written_total`	counter	Bytes written by storage client
`wal_corruption_count`	`firecube_control_plane_corruption_total`	counter	Control-plane WAL corruption events
`wal_torn_tail_recovery_count`	`firecube_control_plane_torn_tail_recovery_total`	counter	Torn WAL tails recovered automatically
`wal_snapshot_rebuild_duration_s`	`firecube_control_plane_snapshot_rebuild_duration_seconds`	gauge	Snapshot rebuild duration
`wal_snapshot_rebuild_count`	`firecube_control_plane_snapshot_rebuild_total`	counter	Snapshot rebuild count
`resume_guard_enforce_duration_s`	`firecube_resume_guard_enforce_duration_seconds`	gauge
`resume_guard_runs_enumerated`	`firecube_resume_guard_runs_enumerated_total`	counter
`resume_guard_spans_scanned`	`firecube_resume_guard_spans_scanned_total`	counter

The telemetry sink may also emit best-effort process memory gauges such as firecube_process_memory_peak_rss_bytes.

Storage Metrics In The Manifest

The ingest JSON manifest includes an engine-owned metrics.storage object when storage information is available.

Field	Meaning
`control_root`	Product-local `.firecube/` directory URI used by ChunkManager
`latest_pointer`	Product-local `LATEST.json` pointer URI
`path`	Final output path
`bytes`	Bytes written
`files`	Files written
`duration_s`	Storage write or upload duration
`storage_type`	Storage class such as `local` or `s3`

Plugins should not construct this object manually.

Next Steps

Observability — metrics, logs, and traces model
Metrics — metric usage and plugin emission
Plugin Observability — plugin telemetry rules