Configuration Reference

Firecube has two configuration surfaces:

Command flags such as --target, --storage-type, --storage-driver, --product-name, --write-mode, and --input-data. These are documented from the live Click command tree in CLI Reference.
Config dataclasses used for storage settings and --option key=value ingestion settings. The schemas below are generated from the actual Python dataclasses.

Pass the required ingest command flags explicitly for each run. For ingest, --storage-type and --storage-driver are inferred from the URI scheme by default unless overridden.

Configuration File

By default, Firecube looks for ~/.config/firecube/config.toml. Use the root --config-file option to select another file:

firecube --config-file /path/to/config.toml ingest <plugin> --show-options

Example:

[storage]
endpoint_url = "https://your-s3-endpoint"
path_style = true
# driver = "fsspec"  # or "obstore" (requires firecube[obstore])

[metrics]
pushgateway_url = "http://localhost:9091"
label_allowlist = ["frp_variant"]

[plugins.my_plugin]
zarr_chunk_shape = '{"timestamp":1,"ny":550,"nx":475}'
zarr_compression = false
zarr_consolidate = false
pipeline_workers = 2
pipeline_batch_size = 40
resume_existing = true

[storage].type and FIRECUBE_STORAGE_TYPE apply only to commands that do not take a product URI (e.g. chunks/*); for URI-bearing commands the storage type comes from the URI scheme or an explicit --storage-type.

[plugins.<name>] only applies when that plugin is installed in the current environment. Use these commands to inspect the effective keys for a plugin:

firecube ingest <plugin_name> --show-options
firecube plugins describe <plugin_name>

Observability environment variables and [metrics] keys are listed in Observability Reference.

Precedence

For storage settings loaded through StorageConfig, precedence is:

CLI overrides
Environment variables
config.toml
Built-in defaults

For firecube ingest, --target and --write-mode remain explicit. Pass --input-data when the plugin reads command-supplied input, and pass --product-name when the plugin has no PRODUCT_NAME or the run needs an override.

Generated Schemas

StorageConfig

Public import: from firecube.core.api import StorageConfig

Global storage configuration for the service.

Location-specific URI fields are intentionally excluded here; runtime storage code now consumes the trimmed config plus a separate target URI boundary where needed.

Attributes:

Name	Type	Description
`storage_type`	`str`	Storage locality/class. Use `"local"` or `"s3"`.
`endpoint_url`	`str \| None`	Optional S3-compatible endpoint URL.
`access_key`	`str \| None`	Optional access key for explicit S3 credentials.
`secret_key`	`str \| None`	Optional secret key for explicit S3 credentials.
`region`	`str \| None`	Optional S3 region name.
`path_style`	`bool`	Whether to use S3 path-style addressing.
`storage_driver`	`str`	Storage I/O driver. Use `"fsspec"` or `"obstore"`.

EngineConfig

Public import: from firecube.ingestor.api import EngineConfig

These fields are accepted as common --option key=value settings and plugin defaults under [plugins.<name>].

Configuration for the Firecube Engine (Runtime & Workspace).

These options control HOW the ingestion runs, not WHAT product logic is applied.

Attributes:

Name	Type	Description
`pipeline_workers`	`int`	Number of pipeline worker threads. A value of 2 or more selects parallel pipeline execution; 1 (the default) runs batches sequentially. Must be at least 1.
`pipeline_batch_size`	`int`	Number of source items per batch. Must be at least 1.
`extract_workers`	`int`	Number of parallel archive-extraction workers used by plugins that extract a batch of source archives (for example ZIPs) to disk before decoding. Extraction is disk-bound and claims no slots, so this is independent of `pipeline_workers`; when both are above 1 the two multiply. Must be at least 1.
`cleanup_workspace`	`bool`	Delete temporary workspace files after the run.
`workspace`	`str \| None`	Optional workspace directory override.
`include_patterns`	`list[str] \| None`	Optional file patterns for source discovery.
`write_mode`	`str`	Write strategy, either `"staged"` or `"direct"`.
`resume_existing`	`bool`	Continue a compatible incomplete or overlapping run.
`force_reingest`	`bool`	Re-process existing spans intentionally.
`incremental`	`bool`	Reserved incremental-mode switch.
`dry_run`	`bool`	Build and validate the run without committing writes.
`duckdb_persist_batches`	`bool`	Persist intermediate DuckDB batch data.
`upload_workers`	`int`	Number of staged upload workers.
`no_progress`	`bool`	Disable progress logging.
`validate_zarr`	`bool`	Validate Zarr state as part of resume checks.
`validate_zarr_group`	`str`	Zarr group to validate when validation is enabled.
`validate_zarr_timeout_s`	`int \| None`	Optional Zarr validation timeout in seconds.
`validate_zarr_max_chunks`	`int \| None`	Optional validation chunk-scan limit.
`validate_zarr_on_timeout`	`str`	Timeout behavior, usually `"warn"`.
`skip_preflight`	`bool`	Skip storage preflight checks.
`slot_start`	`int \| None`	First slot index for orchestrated parallel ingestion.
`slot_end`	`int \| None`	One-past-last slot index for orchestrated parallel ingestion.
`slot_size`	`int \| None`	Slot width used when deriving slot ranges from the environment.
`slot_group`	`str \| None`	Zarr group owned by this worker in multi-group slot runs.
`suppress_static_emission_for_non_owner`	`bool`	Skip static writes in slot-range workers whose `slot_start` does not match `static_owner_slot_start`.
`static_owner_slot_start`	`int \| None`	V1 scalar static-owner slot start for one group per run; required when static suppression is enabled.

TemplateConfig

Public import: from firecube.ingestor.api import TemplateConfig

Base class of the template config tier. Each template declares its config dataclass through template_config_class; the dataclass fields become validated --option key=value settings.

Base class for the template-owned configuration tier.

Each ingestor template declares a template_config_class (e.g. ZarrTemplateConfig for GenericZarrIngestor); the dataclass fields of that class become the validated, typed options accepted for the template. Instances are built from raw caller options via from_options, which rejects unknown keys and coerces values to the annotated field types.

ZarrTemplateConfig

Public import: from firecube.ingestor.api import ZarrTemplateConfig

Bases: TemplateConfig

Configuration for GenericZarrIngestor.

Attributes:

Name	Type	Description
`zarr_chunk_shape`	`dict[str, int] \| None`	Optional per-dimension inner chunk sizes.
`zarr_sharding`	`bool`	Enable Zarr v3 sharding.
`zarr_shard_shape`	`dict[str, int] \| None`	Optional per-dimension shard sizes.
`zarr_compression`	`bool`	`True` (default) delegates to zarr's default codec pipeline (`ZstdCodec(level=0)` in zarr-python v3, matching the upstream default and pre-PR-25 firecube effective behavior). `False` explicitly disables compression. String values are rejected at construction time; only bool values are accepted.
`zarr_codecs`	`list[dict] \| None`	Optional flat list of codec entries in Zarr v3 metadata format `[{"name": str, "configuration": dict}]`. Requires `zarr_compression=True`. When set, the declared pipeline is used instead of zarr's default. Structural validation happens here; codec-specific resolution happens later.
`zarr_consolidate`	`bool`	Consolidate Zarr metadata after writes.
`zarr_time_encoding`	`str \| None`	Optional time encoding override.
`zarr_async_concurrency`	`int`	Async write concurrency used by Zarr.
`zarr_region_write_concurrency`	`int`	Region write concurrency used by Zarr.
`zarr_write_empty_chunks`	`bool`	Pass through Zarr's array write-empty-chunks policy for scoped write phases. Defaults to `False`.
`dask_scheduler`	`str \| None`	Optional Dask scheduler override.
`dask_write_threads`	`int`	Optional write-thread count for Dask-backed writes.

ParquetTemplateConfig

Public import: from firecube.ingestor.api import ParquetTemplateConfig

The current default Parquet writer does not apply parquet_partition_by or parquet_row_group_size. Do not configure these fields until the corresponding writer support is implemented.

Bases: TemplateConfig

Configuration for GenericParquetIngestor.

Attributes:

Name	Type	Description
`parquet_partition_by`	`list[str] \| None`	Optional Hive-style partition columns.
`parquet_row_group_size`	`int \| None`	Optional rows per Parquet row group.

TensogramTemplateConfig

Public import: from firecube.ingestor.api import TensogramTemplateConfig

Bases: TemplateConfig

Configuration for GenericTensogramIngestor.

Attributes:

Name	Type	Description
`tensogram_compression`	`str`	Compression codec for Tensogram output.
`tensogram_message_granularity`	`str`	Message grouping strategy.
`tensogram_allow_nan`	`bool`	Permit NaN values in archive output.
`tensogram_allow_inf`	`bool`	Permit infinite values in archive output.

PluginConfig

Public import: from firecube.ingestor.api import PluginConfig

Plugin authors subclass this dataclass to declare product-specific options.

Base configuration for all Firecube ingestors.

Subclasses should define fields as dataclass fields. Use from_options to parse and validate a raw dictionary.

Attributes:

Name	Type	Description
`_allow_unknown`	`bool`	Class-level escape hatch for plugins that intentionally accept unknown option keys. Keep this `False` for strict validation.

Next Steps

CLI Reference — complete command flags
Configuration Model — precedence and runtime model
Observability Reference — metrics and environment variables