Context & Results
This reference covers the context object passed to plugin hooks, the storage
sessions it exposes, batch inputs, and the result and metrics types. Import
all symbols from firecube.ingestor.api.
| Name | Description |
|---|---|
PluginContext |
Read-only per-run context handed to every plugin-facing hook. |
StorageContext |
Storage sessions for an ingestion run, keyed by role. |
PipelineBatch |
A batch of data ready for processing in the pipeline. |
PipelineRunState |
Immutable snapshot of a pipeline run passed to lifecycle hooks. |
IngestResult |
Final result of one ingestion run. |
PipelineResult |
Result from a pipeline batch processing. |
OutputPaths |
Shared outputs shape used by both batch- and run-level results. |
ResultMetrics |
Shared metrics shape used by both batch- and run-level results. |
PipelineMetrics |
Typed pipeline processing metrics reported in results. |
StorageMetrics |
Typed summary of storage output produced by a run or batch. |
PluginContext
Read-only per-run context handed to every plugin-facing hook.
The engine builds one instance per run from the caller's
IngestContext plus its own runtime state, and passes it to
plugin hooks such as discover_source_files, get_batch_groups,
and build_dataset. Plugin authors never construct it themselves.
All members are read-only. In particular, options is a detached
immutable copy taken when the context is created, and attribute access
outside the documented surface raises AttributeError.
Attributes
property
Input data location for this run, as provided by the caller.
A local path or remote URI (e.g. the --input-data CLI option).
The default discover_source_files implementation scans it for
input files; plugins with custom discovery may interpret it freely.
property
Output target URI for this run, or None when not provided.
Taken from the caller's IngestContext.target. The engine
resolves the actual store location from it together with the
configured write mode, so plugins should treat it as declarative
rather than writing to it directly.
property
Whether intermediate processing should stay in memory.
Set by the caller (e.g. the --in-memory CLI flag). When true,
plugins that stage data (such as DuckDB-backed plugins) should use
in-memory state instead of files under temp_root.
property
Requested output format for this run, or None when unset.
One of "zarr", "parquet", or "tensogram" for the
built-in templates; the CLI defaults it to "zarr".
property
Storage sessions bound to this run, keyed by role, or None.
When output storage is bound, storage.output is the session for
the run's output target. See StorageContext.
property
Per-run workspace directory, or None when no workspace exists.
Created by the engine at run start. Materialized source files and other temporary artifacts live under this root; it may be deleted when the run finishes, so nothing durable should be stored here.
property
Whether this run is allowed to overwrite existing slice data.
Engine-owned flag (e.g. --option force_reingest=true); read-only
for plugins.
property
Whether plugin logic should prefer incremental update behavior.
Engine-owned flag; read-only for plugins. When true, plugins that support it should update existing output instead of reprocessing from scratch.
property
Whether side-effecting writes should be suppressed when supported.
Engine-owned flag; read-only for plugins. Plugins that honor it should skip persistent writes while still exercising the rest of their processing.
property
Telemetry sink for this run, or None when not configured.
Plugins should record metrics and tracing spans only through this
object (emit(), span()) instead of importing a telemetry
backend directly.
property
Plugin options for this run as an immutable mapping.
A detached copy of the caller's options (including --option
CLI overrides) wrapped in types.MappingProxyType when the
context is created: it cannot be mutated, and later changes to
engine state are not reflected in it.
Methods:
Return a single option value with a fallback.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
key
|
str
|
Option name as passed by the caller (e.g. via |
required |
default
|
Any
|
Value returned when the option is not set. |
None
|
Returns:
| Type | Description |
|---|---|
Any
|
The value from |
Ensure a source file is available locally and return its path.
Remote sources (e.g. S3 URIs) are downloaded into the per-run cache
under temp_root; already-local paths are returned directly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
source
|
Any
|
A local path, URI string, or source-file object. |
required |
Returns:
| Type | Description |
|---|---|
Path
|
Local filesystem path to the materialized file. |
Raises:
| Type | Description |
|---|---|
RuntimeError
|
If no materializer is configured for the run and the source cannot be resolved to an existing local file. |
Examples:
Resolve discovered items before handing them to a reader that only accepts local paths:
def build_dataset(self, group, items, ctx):
paths = [ctx.materialize(item) for item in items]
return xr.open_mfdataset(paths, combine="by_coords")
StorageContext
dataclass
Storage sessions for an ingestion run, keyed by role.
Each field binds one storage role to a session that the engine creates
from the resolved target and selected storage driver. Plugins reach it
through PluginContext.storage and should treat the bound
sessions as engine-owned handles: use them to inspect the product
identity and perform storage operations, but do not replace or close
them.
Attributes:
| Name | Type | Description |
|---|---|---|
output |
StorageSession | None
|
Session bound to the run's output target, or |
Batch Input
GenericParquetIngestor and DirectZarrIngestor receive a PipelineBatch.
GenericZarrIngestor receives the batch's source items directly.
dataclass
A batch of data ready for processing in the pipeline.
Run State
dataclass
Immutable snapshot of a pipeline run passed to lifecycle hooks.
Captures the planned batches, worker/batch-size configuration, timing
counters, and (once available) per-batch results and aggregated totals.
Instances are frozen: hooks such as on_pipeline_start observe the
state but cannot mutate it.
Results
dataclass
Final result of one ingestion run.
Aggregates the run's output locations (outputs) and merged metrics
(metrics) after all batches have completed. When output storage is
bound, the engine additionally records the storage completion outcome
(storage_result), the effective write mode (write_mode_applied),
and a manifest summary (manifest) before returning the result to
the caller.
dataclass
dataclass
Metrics
dataclass
Shared metrics shape used by both batch- and run-level results.
Methods:
Update a compatibility metric entry and keep typed fields in sync.
Update compatibility metrics from another mapping or keyword args.
dataclass
Typed pipeline processing metrics reported in results.
Attributes:
| Name | Type | Description |
|---|---|---|
duration_pipeline_s |
float
|
Time spent in batch processing, in seconds. |
rows_processed |
int | None
|
Rows read or processed, when known. |
rows_ingested |
int | None
|
Rows actually written to the output, when known. |
coverage |
list[SpanCoverage]
|
Span coverage entries describing which groups, arrays, and time ranges the run wrote. |
duration_upload_s |
float
|
Time spent uploading staged output, in seconds. |
duration_total_s |
float
|
Total wall-clock duration, in seconds. |