Skip to content

Extensions

This reference covers firecube.ingestor.extensions, the optional helper surface plugins may import alongside firecube.ingestor.api. HEALPix helpers require the firecube[healpix] extra.

Name Description
HealpixBinner Precomputed bin mapping from an input lat/lon grid to HEALPix cells.
build_healpix_binner Precompute the mapping from an input lat/lon grid to HEALPix cells.
cells_in_bbox Return the sorted HEALPix cell ids covering a lon/lat bounding box.
grid_data_to_healpix Grid irregular satellite data onto HEALPix cells (no interpolation).
grid_xarray_dataset_to_healpix Grid multiple variables from an xarray Dataset onto HEALPix cells.
LatLonBinner Precomputed bin mapping from an input grid to a regular lat/lon grid.
build_latlon_binner Precompute the mapping from an input lat/lon grid to regular bins.
grid_data_to_latlon Grid irregular satellite data onto a regular lat/lon grid (no interpolation).
grid_xarray_dataset Grid multiple variables from an xarray Dataset onto a regular lat/lon grid.
aggregate_by_position Scatter flattened values onto a 1-D target axis of length n_targets.
DuckDbMixin Mixin that manages a thread-local DuckDB connection for batch processing.

HEALPix Regridding

firecube.ingestor.extensions.HealpixBinner dataclass

Precomputed bin mapping from an input lat/lon grid to HEALPix cells.

Attributes

n_cells property 

n_cells

Number of cells on the target axis.

npix property 

npix

Total number of HEALPix cells on the sphere at this depth.

firecube.ingestor.extensions.build_healpix_binner 

build_healpix_binner(
    *,
    lat,
    lon,
    depth,
    indexing_scheme="nested",
    ellipsoid="WGS84",
    target_cells=None,
)

Precompute the mapping from an input lat/lon grid to HEALPix cells.

Parameters:

Name Type Description Default
lat ndarray

Latitude of the input samples, in degrees. Either a 1-D coordinate vector (treated as a rectilinear grid) or an N-D array of point coordinates matching lon.

 required
lon ndarray

Longitude of the input samples, in degrees. Same shape rules as lat.

 required
depth int

HEALPix order/level [0, 29]. Determines the cell size (12 * 4**depth cells on the sphere).

 required
indexing_scheme str

HEALPix scheme. Only "nested" is supported.

 'nested'
ellipsoid str

Reference ellipsoid for the lat/lon -> cell mapping (e.g. "WGS84", "sphere"). Two grids share cell ids only when depth and ellipsoid match.

 'WGS84'
target_cells ndarray | None

Optional fixed cell axis (sorted unique cell ids). When given, the axis is exactly these cells and input points outside them are dropped; when None, the axis is derived as the unique cells the input touches.

 None

firecube.ingestor.extensions.cells_in_bbox 

cells_in_bbox(
    *,
    lon_min,
    lat_min,
    lon_max,
    lat_max,
    depth,
    ellipsoid="WGS84",
    step=None,
)

Return the sorted HEALPix cell ids covering a lon/lat bounding box.

Useful as the target_cells axis for a regional product: enumerate the footprint once, then bin every granule/timestep onto it.

The box is sampled on a regular lon/lat mesh and each sample mapped to its cell; the unique set is returned. step (degrees) defaults to half the cell size at depth so every interior cell is hit. This handles boxes of any size (HEALPix FOV-coverage queries are limited to ~90° per side).

firecube.ingestor.extensions.grid_data_to_healpix 

grid_data_to_healpix(
    *,
    lat,
    lon,
    data,
    depth,
    variable_name="value",
    indexing_scheme="nested",
    ellipsoid="WGS84",
    aggregation="mean",
    fill_value=None,
    target_cells=None,
)

Grid irregular satellite data onto HEALPix cells (no interpolation).

firecube.ingestor.extensions.grid_xarray_dataset_to_healpix 

grid_xarray_dataset_to_healpix(
    *,
    ds,
    lat_var,
    lon_var,
    data_vars,
    depth,
    indexing_scheme="nested",
    ellipsoid="WGS84",
    aggregation="mean",
    fill_value=None,
    target_cells=None,
    cell_dim=DEFAULT_CELL_DIM,
    cell_coord=DEFAULT_CELL_COORD,
)

Grid multiple variables from an xarray Dataset onto HEALPix cells.

Lat/Lon Regridding

firecube.ingestor.extensions.LatLonBinner dataclass

Precomputed bin mapping from an input grid to a regular lat/lon grid.

firecube.ingestor.extensions.build_latlon_binner 

build_latlon_binner(
    *, lat, lon, grid_spacing, bounds=None, fill_value=None
)

Precompute the mapping from an input lat/lon grid to regular bins.

Parameters:

Name Type Description Default
lat ndarray

Input geolocation latitude, a 1-D coordinate vector or an N-D array matching lon.

 required
lon ndarray

Input geolocation longitude, a 1-D coordinate vector or an N-D array matching lat.

 required
grid_spacing float

Target cell size in degrees (> 0).

 required
bounds tuple[float, float, float, float] | None

Optional (lat_min, lat_max, lon_min, lon_max) fixing the output grid extent. When given, the axis is exactly these bounds and input points outside them are dropped -- pin one set of bounds to reuse the same grid across granules. When None, the extent is derived from the data (floor(min)/ceil(max)).

 None
fill_value float | None

Accepted for signature parity; data masking is applied by callers.

 None

firecube.ingestor.extensions.grid_data_to_latlon 

grid_data_to_latlon(
    *,
    lat,
    lon,
    data,
    grid_spacing=0.1,
    variable_name="value",
    fill_value=None,
    aggregation="mean",
    bounds=None,
)

Grid irregular satellite data onto a regular lat/lon grid (no interpolation).

firecube.ingestor.extensions.grid_xarray_dataset 

grid_xarray_dataset(
    *,
    ds,
    lat_var,
    lon_var,
    data_vars,
    grid_spacing=0.1,
    fill_value=None,
    aggregation="mean",
    bounds=None,
)

Grid multiple variables from an xarray Dataset onto a regular lat/lon grid.

Positional Binning

firecube.ingestor.extensions.aggregate_by_position 

aggregate_by_position(
    *,
    values_flat,
    valid_point_index,
    position,
    n_targets,
    aggregation,
)

Scatter flattened values onto a 1-D target axis of length n_targets.

Parameters:

Name Type Description Default
values_flat ndarray

Flattened input values (same flattening as the binner was built from).

 required
valid_point_index ndarray

Indices into values_flat for the points the binner kept.

 required
position ndarray

For each kept point, the target-axis index in [0, n_targets).

 required
n_targets int

Length of the output axis.

 required
aggregation str

One of AGGREGATIONS. any returns a uint8 presence mask; all others return float32 with NaN in empty cells.

 required

DuckDB Support

DuckDbMixin adds a per-batch DuckDB connection to a plugin through the cooperative batch_setup/batch_teardown lifecycle hooks.

firecube.ingestor.extensions.DuckDbMixin

Mixin that manages a thread-local DuckDB connection for batch processing.

Why thread-local? In parallel pipeline mode, each worker thread calls batch_setup / batch_teardown independently. DuckDB connections are not thread-safe for concurrent writes; giving each thread its own in-memory connection avoids locking and eliminates the risk of file-level write conflicts when using a persistent *.duckdb file.

Lifecycle per batch (driven by BaseIngestor): batch_setup(ctx) → opens connection, applies settings, calls prepare_duckdb_schema hook. batch_teardown(ctx) → closes and deletes the thread-local connection.

MRO cooperation

Both batch_setup and batch_teardown call super() cooperatively so that this mixin can be stacked with other mixins (e.g. ProductDataMixin) without silently dropping their setup logic.

Persistent mode (duckdb_persist_batches=true): Workers use an in-memory DB by default. Persistence can be enabled for debugging or when accumulating rows across batches before a single final export. Persistent connections use a file under the run's temp workspace; the main thread pre-initialises the schema in GenericZarrIngestor.on_pipeline_start to avoid races on first write.

Methods:

setup_duckdb 

setup_duckdb(workspace=None, options=None, in_memory=True)

Initialize DuckDB connection with settings (THREAD-LOCAL).

teardown_duckdb 

teardown_duckdb()

Close DuckDB connection (THREAD-LOCAL).

prepare_duckdb_schema 

prepare_duckdb_schema(con, ctx)

Optional hook to initialize tables in persistent DB mode.

See Also