Skip to content

NetCDF To Zarr: Observability

Goal

Add one domain-specific metric to the plugin from the first tutorial, run ingestion with a Prometheus Pushgateway, and inspect the exported metric.

Continue from NetCDF To Zarr. You will keep the same plugin and add one metric through ctx.telemetry.

Prerequisites

  • The completed Quickstart project in the current directory.
  • Docker available with local port 9091 free.

Start A Pushgateway

This tutorial uses Docker to run a temporary local Pushgateway:

docker run --rm -d \
  --name firecube-tutorial-pushgateway \
  -p 9091:9091 \
  prom/pushgateway:v1.11.1

Set its URL for the remaining commands:

export FIRECUBE_PUSHGATEWAY_URL="http://localhost:9091"

Confirm that it is ready:

curl -s "$FIRECUBE_PUSHGATEWAY_URL/-/ready"

Expected output:

OK

Add One Metric

In plugins_dev/firecube-weather-netcdf/src/firecube_weather_netcdf/ingestor.py, add this block near the end of build_dataset, just before return result:

        if ctx.telemetry is not None:
            ctx.telemetry.emit(
                "weather_netcdf_files",
                len(items),
                kind="counter",
                meta={"group": group, "status": "success"},
            )

counter is the right kind here because the value is a count that increases during the run. Firecube exports it as:

firecube_weather_netcdf_files_total

Run The Plugin

PRODUCT_URI="file://$PWD/quickstart-output/weather_netcdf_observed.zarr"

firecube ingest weather_netcdf \
  --input-data quickstart-data/weather-netcdf \
  --target "$PRODUCT_URI" \
  --product-name weather_netcdf_observed \
  --storage-type local \
  --storage-driver fsspec \
  --output-format zarr \
  --write-mode direct

Expected logs on stderr include:

"message":"Found 4 files"

Expected command output on stdout includes:

"plugin": "weather_netcdf"
...
"files_processed": 4
...
"count": 4
"product": "weather_netcdf_observed"

Verify The Product

python - <<'PY'
import xarray as xr

ds = xr.open_zarr(
    "quickstart-output/weather_netcdf_observed.zarr",
    group="default",
    consolidated=False,
)
assert ds.sizes["timestamp"] == 4
print("ok")
PY

Expected output:

ok

Verify The Metric

Firecube flushes buffered metrics at the end of the run. Query the Pushgateway and select the metric added above:

curl -s "$FIRECUBE_PUSHGATEWAY_URL/metrics" \
  | grep "firecube_weather_netcdf_files_total"

Expected output includes a sample with the value 4:

firecube_weather_netcdf_files_total{group="default",status="success",...} 4

If the metric is absent, check the ingestion logs for Failed to push metrics to Pushgateway, then confirm the configured URL is reachable from the Firecube process.

Stop the tutorial Pushgateway when verification is complete:

docker stop firecube-tutorial-pushgateway

Next Steps