Configuration Reference

Firecube has two configuration surfaces:

Command flags such as --target, --storage-type, --storage-driver, --product-name, --write-mode, and --input-data. These are documented from the live Click command tree in CLI Reference.
Config dataclasses used for storage settings and --option key=value ingestion settings. The schemas below are generated from the actual Python dataclasses.

Pass the required ingest command flags explicitly for each run. For ingest, --storage-type and --storage-driver are inferred from the URI scheme by default unless overridden.

Configuration File

By default, Firecube looks for ~/.config/firecube/config.toml. Use the root --config-file option to select another file:

firecube --config-file /path/to/config.toml ingest <plugin> --show-options

Example:

[storage]
endpoint_url = "https://your-s3-endpoint"
path_style = true
# driver = "fsspec"  # or "obstore" (requires firecube[obstore])

[database.duckdb]
duckdb_max_temp_directory_size = "100GiB"
duckdb_memory_limit = "8GB"
duckdb_threads = 4

[metrics]
pushgateway_url = "http://localhost:9091"
label_allowlist = ["frp_variant"]

[plugins.my_plugin]
zarr_chunk_shape = '{"timestamp":1,"ny":550,"nx":475}'
zarr_compression = false
zarr_consolidate = false
pipeline_parallel = true
pipeline_workers = 2
pipeline_batch_size = 40
resume_existing = true

[storage].type and FIRECUBE_STORAGE_TYPE apply only to commands that do not take a product URI (e.g. chunks/*); for URI-bearing commands the storage type comes from the URI scheme or an explicit --storage-type.

[plugins.<name>] only applies when that plugin is installed in the current environment. Use these commands to inspect the effective keys for a plugin:

firecube ingest <plugin_name> --show-options
firecube plugins describe <plugin_name>

Observability environment variables and [metrics] keys are listed in Observability Reference.

Precedence

For storage settings loaded through StorageConfig, precedence is:

CLI overrides
Environment variables
config.toml
Built-in defaults

For firecube ingest, required command flags remain explicit even when storage defaults exist in the config file or environment. Always pass --input-data, --target, --product-name, and --write-mode on ingest.

Generated Schemas

StorageConfig

Public import: from firecube.core.api import StorageConfig

Global storage configuration for the service.

Location-specific URI fields are intentionally excluded here; runtime storage code now consumes the trimmed config plus a separate target URI boundary where needed.

Attributes:

Name	Type	Description
`storage_type`	`str`	Storage locality/class. Use `"local"` or `"s3"`.
`endpoint_url`	`str \| None`	Optional S3-compatible endpoint URL.
`access_key`	`str \| None`	Optional access key for explicit S3 credentials.
`secret_key`	`str \| None`	Optional secret key for explicit S3 credentials.
`region`	`str \| None`	Optional S3 region name.
`path_style`	`bool`	Whether to use S3 path-style addressing.
`storage_driver`	`str`	Storage I/O driver. Use `"fsspec"` or `"obstore"`.

EngineConfig

Public import: from firecube.ingestor.api import EngineConfig

These fields are accepted as common --option key=value settings and plugin defaults under [plugins.<name>].

Configuration for the Firecube Engine (Runtime & Workspace).

These options control HOW the ingestion runs, not WHAT product logic is applied.

Attributes:

Name	Type	Description
`pipeline_parallel`	`bool`	Enable parallel batch preprocessing.
`pipeline_workers`	`int`	Number of pipeline worker threads.
`pipeline_batch_size`	`int`	Number of source items per batch.
`cleanup_workspace`	`bool`	Delete temporary workspace files after the run.
`workspace`	`str \| None`	Optional workspace directory override.
`include_patterns`	`list[str] \| None`	Optional file patterns for source discovery.
`write_mode`	`str`	Write strategy, either `"staged"` or `"direct"`.
`resume_existing`	`bool`	Continue a compatible incomplete or overlapping run.
`force_reingest`	`bool`	Re-process existing spans intentionally.
`incremental`	`bool`	Reserved incremental-mode switch.
`dry_run`	`bool`	Build and validate the run without committing writes.
`duckdb_persist_batches`	`bool`	Persist intermediate DuckDB batch data.
`upload_workers`	`int`	Number of staged upload workers.
`no_progress`	`bool`	Disable progress logging.
`validate_zarr`	`bool`	Validate Zarr state as part of resume checks.
`validate_zarr_group`	`str`	Zarr group to validate when validation is enabled.
`validate_zarr_timeout_s`	`int \| None`	Optional Zarr validation timeout in seconds.
`validate_zarr_max_chunks`	`int \| None`	Optional validation chunk-scan limit.
`validate_zarr_on_timeout`	`str`	Timeout behavior, usually `"warn"`.
`skip_preflight`	`bool`	Skip storage preflight checks.
`slot_start`	`int \| None`	First slot index for orchestrated parallel ingestion.
`slot_end`	`int \| None`	One-past-last slot index for orchestrated parallel ingestion.
`slot_size`	`int \| None`	Slot width used when deriving slot ranges from the environment.
`slot_group`	`str \| None`	Zarr group owned by this worker in multi-group slot runs.

ZarrTemplateConfig

Public import: from firecube.ingestor.api import ZarrTemplateConfig

Bases: TemplateConfig

Configuration for GenericZarrIngestor.

Attributes:

Name	Type	Description
`zarr_chunk_shape`	`dict[str, int] \| None`	Optional per-dimension inner chunk sizes.
`zarr_sharding`	`bool`	Enable Zarr v3 sharding.
`zarr_shard_shape`	`dict[str, int] \| None`	Optional per-dimension shard sizes.
`zarr_compression`	`bool \| str`	Compression setting. `False` disables compression, `True` selects the default codec, and a string selects a codec name.
`zarr_consolidate`	`bool`	Consolidate Zarr metadata after writes.
`zarr_time_encoding`	`str \| None`	Optional time encoding override.
`zarr_async_concurrency`	`int`	Async write concurrency used by Zarr.
`dask_scheduler`	`str \| None`	Optional Dask scheduler override.
`dask_write_threads`	`int`	Optional write-thread count for Dask-backed writes.

ParquetTemplateConfig

Public import: from firecube.ingestor.api import ParquetTemplateConfig

Bases: TemplateConfig

Configuration for GenericParquetIngestor.

Attributes:

Name	Type	Description
`parquet_partition_by`	`list[str] \| None`	Optional Hive-style partition columns.
`parquet_row_group_size`	`int \| None`	Optional rows per Parquet row group.

TensogramTemplateConfig

Public import: from firecube.ingestor.api import TensogramTemplateConfig

Bases: TemplateConfig

Configuration for GenericTensogramIngestor.

Attributes:

Name	Type	Description
`tensogram_compression`	`str`	Compression codec for Tensogram output.
`tensogram_message_granularity`	`str`	Message grouping strategy.
`tensogram_allow_nan`	`bool`	Permit NaN values in archive output.
`tensogram_allow_inf`	`bool`	Permit infinite values in archive output.

PluginConfig

Public import: from firecube.ingestor.api import PluginConfig

Plugin authors subclass this dataclass to declare product-specific options.

Base configuration for all Firecube ingestors.

Subclasses should define fields as dataclass fields. Use from_options to parse and validate a raw dictionary.

Attributes:

Name	Type	Description
`_allow_unknown`	`bool`	Class-level escape hatch for plugins that intentionally accept unknown option keys. Keep this `False` for strict validation.

Next Steps

CLI Reference — complete command flags
Configuration Model — precedence and runtime model
Observability Reference — metrics and environment variables