Configuration Reference

Firecube has two configuration surfaces:

Command flags such as --target, --storage-type, --storage-driver, --product-name, --write-mode, and --input-data. These are documented from the live Click command tree in CLI Reference.
Config dataclasses used for storage settings and --option key=value ingestion settings. The schemas below are generated from the actual Python dataclasses.

Pass the required ingest command flags explicitly for each run. For ingest, --storage-type and --storage-driver are inferred from the URI scheme by default unless overridden.

Configuration File

By default, Firecube looks for ~/.config/firecube/config.toml. Use the root --config-file option to select another file:

firecube --config-file /path/to/config.toml ingest <plugin> --show-options

Example:

[storage]
endpoint_url = "https://your-s3-endpoint"
path_style = true
# driver = "fsspec"  # or "obstore" (requires firecube[obstore])

[metrics]
pushgateway_url = "http://localhost:9091"
label_allowlist = ["frp_variant"]

[plugins.my_plugin]
zarr_chunk_shape = '{"timestamp":1,"ny":550,"nx":475}'
zarr_compression = false
zarr_consolidate = false
pipeline_workers = 2
pipeline_batch_size = 40
resume_existing = true

[storage].type and FIRECUBE_STORAGE_TYPE apply only to commands that do not take a product URI (e.g. chunks/*); for URI-bearing commands the storage type comes from the URI scheme or an explicit --storage-type.

[plugins.<name>] only applies when that plugin is installed in the current environment. Use these commands to inspect the effective keys for a plugin:

firecube ingest <plugin_name> --show-options
firecube plugins describe <plugin_name>

Observability environment variables and [metrics] keys are listed in Observability Reference.

Precedence

For storage settings loaded through StorageConfig, precedence is:

CLI overrides
Environment variables
config.toml
Built-in defaults

For firecube ingest, --target and --write-mode remain explicit. Pass --input-data when the plugin reads command-supplied input, and pass --product-name when the plugin has no PRODUCT_NAME or the run needs an override.

Generated Schemas

StorageConfig

Public import: from firecube.core.api import StorageConfig

Global storage configuration for the service.

Location-specific URI fields are intentionally excluded here; runtime storage code now consumes the trimmed config plus a separate target URI boundary where needed.

Attributes:

Name	Type	Description
`storage_type`	`str`	Storage locality/class. Use `"local"` or `"s3"`.
`endpoint_url`	`str \| None`	Optional S3-compatible endpoint URL.
`access_key`	`str \| None`	Optional access key for explicit S3 credentials.
`secret_key`	`str \| None`	Optional secret key for explicit S3 credentials.
`region`	`str \| None`	Optional S3 region name.
`path_style`	`bool`	Whether to use S3 path-style addressing.
`storage_driver`	`str`	Storage I/O driver. Use `"fsspec"` or `"obstore"`.
`anonymous`	`bool`	Explicit anonymous S3 access. When True and storage type is S3, disables request signing. Never inferred from missing credentials.

EngineConfig

Public import: from firecube.ingestor.api import EngineConfig

See Discover Source Data for input-filter commands and Configuration Model for configuration precedence.

Configuration for the Firecube Engine (Runtime & Workspace).

These options control HOW the ingestion runs, not WHAT product logic is applied.

Attributes:

Name	Type	Description
`pipeline_workers`	`int`	Number of pipeline worker threads. A value of 2 or more selects parallel pipeline execution; 1 (the default) runs batches sequentially. Must be at least 1.
`pipeline_batch_size`	`int`	Number of source items per batch. Must be at least 1.
`extract_workers`	`int`	Number of parallel archive-extraction workers used by plugins that extract a batch of source archives (for example ZIPs) to disk before decoding. Extraction is disk-bound and claims no slots, so this is independent of `pipeline_workers`; when both are above 1 the two multiply. Must be at least 1.
`cleanup_workspace`	`bool`	Delete temporary workspace files after the run.
`workspace`	`str \| None`	Optional workspace directory override.
`input_filters`	`list[str] \| None`	Optional list of input-file filters. Positive entries add matches to built-in discovery; `!` entries exclude matches regardless of order. Matching is case-sensitive; default suffix recognition remains case-insensitive. Supply one JSON list through `--input-filters` or a list in plugin configuration. An explicit CLI list replaces configured filters; `[]` restores built-in selection. Custom discovery hooks must apply this field themselves. See `discover_input_files` for matching and escaping rules.
`write_mode`	`str`	Write strategy, either `"staged"` or `"direct"`.
`resume_existing`	`bool`	Continue a compatible incomplete or overlapping run.
`force_reingest`	`bool`	Re-process existing spans intentionally. Overlapping timestamps are overwritten in-place (region overwrite); new tail timestamps are appended. No separate delete step is required. Refuses inserts, duplicates, and NaT values. Use `chunks delete` first when physical chunk removal is needed.
`incremental`	`bool`	Reserved incremental-mode switch.
`dry_run`	`bool`	Reserved: `ingest` does not implement a dry run yet and writes normally. Use `--dry-run` on maintenance commands (`chunks delete`, `chunks delete-span`, `zarr preallocate`) for previews.
`duckdb_persist_batches`	`bool`	Persist intermediate DuckDB batch data.
`upload_workers`	`int`	Number of staged upload workers.
`no_progress`	`bool`	Disable progress logging.
`allow_empty_source`	`bool`	When `True`, treat an empty source list as a successful no-op rather than raising an error. Defaults to `False` (empty source is an error). Opt-in via `--option allow_empty_source=true`.
`validate_zarr`	`bool`	Validate Zarr state as part of resume checks.
`validate_zarr_group`	`str`	Zarr group to validate when validation is enabled.
`validate_zarr_timeout_s`	`int \| None`	Optional Zarr validation timeout in seconds.
`validate_zarr_max_chunks`	`int \| None`	Optional validation chunk-scan limit.
`validate_zarr_on_timeout`	`str`	Timeout behavior, usually `"warn"`.
`skip_preflight`	`bool`	Skip storage preflight checks.
`slot_start`	`int \| None`	First slot index for orchestrated parallel ingestion.
`slot_end`	`int \| None`	One-past-last slot index for orchestrated parallel ingestion.
`slot_size`	`int \| None`	Slot width used when deriving slot ranges from the environment.
`slot_group`	`str \| None`	Zarr group owned by this worker in multi-group slot runs.
`suppress_static_emission_for_non_owner`	`bool`	Skip static writes in slot-range workers whose `slot_start` does not match `static_owner_slot_start`.
`static_owner_slot_start`	`int \| None`	V1 scalar static-owner slot start for one group per run; required when static suppression is enabled.

TemplateConfig

Public import: from firecube.ingestor.api import TemplateConfig

Base class of the template config tier. Each template declares its config dataclass through template_config_class; the dataclass fields become validated --option key=value settings.

Base class for the template-owned configuration tier.

Each ingestor template declares a template_config_class (e.g. ZarrTemplateConfig for GenericZarrIngestor); the dataclass fields of that class become the validated, typed options accepted for the template. Instances are built from raw caller options via from_options, which rejects unknown keys and coerces values to the annotated field types.

Subclass the matching template config to declare static plugin defaults, then attach it as the plugin's template_config_class. Explicit caller settings override those defaults. Hooks read self.template_config; the default values are not inserted into ctx.options. For dynamic append layouts, override GenericZarrIngestor.get_zarr_config.

ZarrTemplateConfig

Public import: from firecube.ingestor.api import ZarrTemplateConfig

Bases: TemplateConfig

Configuration for GenericZarrIngestor.

Attributes:

Name	Type	Description
`zarr_chunk_shape`	`dict[str, int] \| None`	Optional per-dimension inner chunk sizes.
`zarr_sharding`	`bool`	Enable Zarr v3 sharding.
`zarr_shard_shape`	`dict[str, int] \| None`	Optional per-dimension shard sizes.
`zarr_compression`	`bool`	`True` (default) delegates to zarr's default codec pipeline (`ZstdCodec(level=0)` in zarr-python v3, matching the upstream default and pre-PR-25 firecube effective behavior). `False` explicitly disables compression. String values are rejected at construction time; only bool values are accepted.
`zarr_codecs`	`list[dict] \| None`	Optional flat list of codec entries in Zarr v3 metadata format `[{"name": str, "configuration": dict}]`. Requires `zarr_compression=True`. When set, the declared pipeline is used instead of zarr's default. Structural validation happens here; codec-specific resolution happens later.
`zarr_consolidate`	`bool`	Consolidate Zarr metadata after writes.
`zarr_time_encoding`	`str \| None`	Not implemented. Non-empty values raise `ConfigurationError` during validation. Set time encoding explicitly in the plugin instead.
`zarr_async_concurrency`	`int`	Async write concurrency used by Zarr.
`zarr_region_write_concurrency`	`int`	Region write concurrency used by Zarr.
`zarr_write_empty_chunks`	`bool`	Pass through Zarr's array write-empty-chunks policy for scoped write phases. Defaults to `False`.
`dask_scheduler`	`str \| None`	Optional Dask scheduler override.
`dask_write_threads`	`int`	Optional write-thread count for Dask-backed writes.

ParquetTemplateConfig

Public import: from firecube.ingestor.api import ParquetTemplateConfig

The current default Parquet writer does not apply parquet_partition_by or parquet_row_group_size. Do not configure these fields until the corresponding writer support is implemented.

Bases: TemplateConfig

Configuration for GenericParquetIngestor.

Attributes:

Name	Type	Description
`parquet_partition_by`	`list[str] \| None`	Optional Hive-style partition columns.
`parquet_row_group_size`	`int \| None`	Optional rows per Parquet row group.

TensogramTemplateConfig

Public import: from firecube.ingestor.api import TensogramTemplateConfig

Bases: TemplateConfig

Configuration for GenericTensogramIngestor.

Attributes:

Name	Type	Description
`tensogram_compression`	`str`	Compression codec for Tensogram output.
`tensogram_message_granularity`	`str`	Message grouping strategy.
`tensogram_allow_nan`	`bool`	Permit NaN values in archive output.
`tensogram_allow_inf`	`bool`	Permit infinite values in archive output.

PluginConfig

Public import: from firecube.ingestor.api import PluginConfig

Plugin authors subclass this dataclass to declare product-specific options.

Base configuration for all Firecube ingestors.

Decorate subclasses that add fields with @dataclass and attach the class to the plugin as plugin_config_class. Hooks read the validated instance through self.plugin_config. Explicit caller options override dataclass defaults, including explicit values equal to a core default.

Engine and template settings are available through self.engine_config and self.template_config respectively. PluginContext.options is an immutable copy of runtime options, including supplied values and engine-added values such as run_id. It does not contain every typed default; ctx.option(key, default) uses its fallback for absent keys. Use from_options to parse and validate a raw dictionary outside a run.

Attributes:

Name	Type	Description
`_allow_unknown`	`bool`	Class-level escape hatch for plugins that intentionally accept unknown option keys. Keep this `False` for strict validation.

Next Steps

CLI Reference — complete command flags
Configuration Model — precedence and runtime model
Observability Reference — metrics and environment variables