Firecube Zarr products in Earth observation AI-pedia

A Firecube Zarr product is an analysis-ready multidimensional datacube store generated from Earth observation satellite instruments. The products are produced by Firecube, a specialised batch ingestion framework and command-line interface developed by the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT). Hosted natively within the European Weather Cloud, Firecube transforms discrete, granular satellite archives into unified, cloud-optimised array stores formatted according to the open-source Zarr storage specification.

Historically, satellite observation data from missions such as Meteosat Second Generation (MSG), Meteosat Third Generation (MTG), and the EUMETSAT Polar System - Second Generation (EPS-SG) have been distributed as individual files called granules. These granules are packaged in complex hierarchical formats, including Hierarchical Data Format version 5 (HDF5), Network Common Data Form version 4 (netCDF-4), or native binary formats like High Rate Information Transmission (HRIT). While suitable for file-based archival, analyzing time-series or regional phenomena across thousands of separate granules requires users to repeatedly download, decompress, and stitch files. This process creates significant Input/Output (I/O) bottlenecks and high compute overhead.

Firecube Zarr products address this operational inefficiency by reorganising petabyte-scale observations into continuous, N-dimensional regular grids along logical dimensions such as time, spectral channel, and spatial coordinates. Stored on S3-compatible cloud object storage systems, these products allow scientists to perform serverless, byte-range queries directly over the cloud. Using scientific Python libraries such as xarray and Dask, users can stream targeted geographic bounding boxes or long-term temporal series without downloading or indexing full satellite swath files.

Background and Operational Context

EUMETSAT is an intergovernmental organisation responsible for operating Europe's meteorological satellites. Its systems deliver continuous operational streams of Earth observation data to national meteorological services, environmental monitoring agencies, and scientific researchers. The operational cadence of these satellite platforms produces massive volumes of observational data every 24 hours.

Satellite Missions and Data Characteristics

The primary instruments feeding EUMETSAT processing chains include:

• SEVIRI (Spinning Enhanced Visible and InfraRed Imager): Mounted on Meteosat Second Generation (MSG) geostationary satellites, SEVIRI captures a full-disk scan of the Earth every 15 minutes across 12 spectral bands, with a spatial resolution of 1 to 3 kilometres at the sub-satellite point. Rapid-scanning services capture subsets of the Northern Hemisphere every 5 minutes.
• FCI (Flexible Combined Imager): Deployed on Meteosat Third Generation (MTG) satellites, the FCI improves upon SEVIRI by offering 16 spectral channels, an increased full-disk scanning frequency of 10 minutes, and spatial resolutions between 0.5 and 2 kilometres. This instrument increases data production rates by nearly an order of magnitude compared to MSG.
• LI (Lightning Imager): Also on MTG, the LI continuously measures optical pulses triggered by lightning discharges, producing high-frequency, event-based spatial point and grid observations.
• EPS-SG Instruments: Polar-orbiting spacecraft carrying sensors such as METimage and the Infrared Atmospheric Sounding Interferometer - New Generation (IASI-NG), which acquire continuous global swaths with non-uniform, trajectory-dependent geometry.

In their raw and native Level-1B (calibrated radiances) and Level-2 (derived geophysical products) states, these instruments produce tens of millions of discrete files per year. A 10-year climatic analysis of SEVIRI observations requires accessing roughly 350,000 individual files. For MTG FCI, the number of files and total byte volume are considerably higher.

The European Weather Cloud

To support the compute demands of these data volumes, EUMETSAT and the European Centre for Medium-Range Weather Forecasts (ECMWF) established the European Weather Cloud (EWC). Built on open-source cloud computing platforms (primarily OpenStack) and software-defined storage architectures (predominantly Ceph Object Gateway providing Amazon S3-compatible APIs), the EWC provides compute nodes located adjacent to satellite ingest archives.

Traditional file formats create major performance challenges when deployed on cloud object storage. File formats such as netCDF-4 and HDF5 were architected for local, POSIX-compliant file systems where file opening, metadata parsing, and small-offset byte seeks have microsecond latencies. When stored on object storage, every metadata read or internal file-pointer jump requires an HTTP GET request, incurring latencies of 10 to 50 milliseconds. Consequently, attempting to read a slice of data spanning thousands of netCDF files over an S3 interface can require hours just to open the files and parse headers, before reading any science data.

Architecture of the Firecube Ingestion Engine

Firecube was created to solve the performance limitations of granule-based storage on the European Weather Cloud. It is an automated, plugin-driven batch ingestion command-line interface (CLI) written in Python. It continuously or batch-transforms operational satellite file feeds into consolidated, cloud-optimised Zarr stores.

Plugin-Based Framework

Firecube relies on a decoupled, modular plugin design to handle different satellite instruments and processing levels. Each plugin contains the specialized logic needed to read native satellite formats, extract geolocation coordinates, perform radiometric calibration, apply quality masks, and map irregular instrument structures to standardized multidimensional grids.

The ingestion lifecycle inside a Firecube plugin runs through a pipeline of discrete stages:

1. Discovery: Locating newly produced or archived granules via catalog APIs, filesystem scans, or messaging brokers.
2. Extraction and Parsing: Opening native inputs (e.g., HRIT compressed segments, raw binary packages, or netCDF-4 granules) and reading raw digital counts and operational flags.
3. Standardisation: Converting digital counts to physical units (such as top-of-atmosphere radiance, reflectance, or brightness temperature) using dynamic calibration coefficients extracted from granule metadata.
4. Coordinate Rectification: Calculating or assigning continuous spatial coordinates (e.g., geostationary projection coordinates or regular latitude/longitude grids) and assigning standardized ISO-8601 temporal coordinate indices.
5. Partition Allocation: Interacting with the Firecube ChunkManager to obtain chunk-index boundaries for the current batch.
6. Array Persistence: Writing data payloads into predefined target Zarr chunks stored on Ceph S3 object storage.

The ChunkManager Component

A critical architectural element of Firecube is its ChunkManager. In standard open-source geospatial tools, appending data to a Zarr store concurrently from multiple worker processes frequently leads to race conditions. These collisions can corrupt the global metadata file (.zarray) or cause workers to overwrite one another's partial chunk boundaries.

Firecube's ChunkManager separates metadata management from data chunk writing. It structures ingestion work into abstract units referred to as spans and runs:

• Span: A deterministic range across the multidimensional coordinate system, most commonly a bounded continuous interval along the temporal axis (for example, a 24-hour window spanning time indices [0:96] for 15-minute MSG data).
• Run: A tracked operational attempt by an ingestion worker to populate a specific span. The ChunkManager monitors active runs, prevents duplicate workers from claiming identical coordinate spans, and guarantees that chunks are written atomically.

By using the ChunkManager to coordinate workers, Firecube can execute massively parallel ingestion jobs across hundreds of compute nodes on the European Weather Cloud without requiring continuous, distributed file-system locks.

Structural Specification of Firecube Zarr Stores

Zarr is an open-source specification for storing chunked, compressed, N-dimensional arrays. Rather than holding an entire dataset within a single monolithic file, Zarr stores the dataset as a logical group hierarchy composed of lightweight configuration files and separate compressed binary objects for each data chunk.

Directory and Object Layout

A Firecube Zarr product on an S3-compatible object store exists as an S3 bucket or prefix (folder-like key hierarchy). The root prefix contains metadata files, coordinate vectors, and data variables:

firecube-msg-seviri-hrv.zarr/
├── .zgroup
├── .zattrs
├── time/
│   ├── .zarray
│   ├── .zattrs
│   └── 0
├── channel/
│   ├── .zarray
│   ├── .zattrs
│   └── 0
├── y/
│   ├── .zarray
│   ├── .zattrs
│   └── 0
├── x/
│   ├── .zarray
│   ├── .zattrs
│   └── 0
├── radiance/
│   ├── .zarray
│   ├── .zattrs
│   ├── 0.0.0.0
│   ├── 0.0.0.1
│   ├── 1.0.0.0
│   └── 1.0.0.1
└── data_quality_flag/
    ├── .zarray
    ├── .zattrs
    ├── 0.0.0.0
    └── 1.0.0.0

Within this hierarchy, the .zgroup file is a JSON document declaring that the store represents an array group. The .zattrs files contain user and system metadata, such as instrument identifiers, processing history, and standard attribute conventions. The .zarray files describe array dimensions, data types, fill values, compression codecs, and chunk shapes. The numeric files (e.g., 0.0.0.0) represent the actual compressed data chunks, indexed by their position along each dimension.

Dimensional Layout and Indexing

Firecube organises satellite variables along standardized logical dimensions. For geostationary instruments like SEVIRI and FCI, the standard dimensional order is:

(time, channel, y, x)

For Level-2 geophysical variables that have already been mapped to standard geographical coordinates or for polar missions gridded onto regular surfaces, the layout uses:

(time, latitude, longitude)

Spatial coordinates along x and y for geostationary data are typically preserved in the native geostationary projection projection plane (defined by the satellite's nominal orbital sub-satellite point, sweep-angle axis, and perspective height). Retaining the native projection avoids interpolation errors and spatial smearing that occur when raw satellite radiances are reprojected onto an equirectangular grid during ingestion.

Chunk Geometry and Optimization

The selection of chunk shape is a fundamental performance decision in Zarr datacube design. In an S3 object store, every chunk retrieval incurs an HTTP overhead. If chunks are too small (e.g., less than 1 megabyte), an analysis over a large spatial or temporal domain requires thousands of individual network requests, causing request overhead to dominate execution time. Conversely, if chunks are too large (e.g., greater than 200 megabytes), reading a localized subset transfers unnecessary data across the network, wasting memory and network bandwidth.

Firecube aims for compressed chunk sizes between 10 megabytes and 64 megabytes. Because Earth observation workflows generally split into either long-term time-series extraction at specific points or broad spatial analysis across small time windows, Firecube balances spatial and temporal chunk shapes based on product requirements:

Product Class	Primary Analysis Pattern	Representative Chunk Shape (time, y, x)	Approximate Uncompressed Size
Operational Rapid Scan (e.g., FCI RSS)	Temporal tracking of severe convection	(12, 500, 500)	12 MB (Float32, 1 channel)
Full Disk Climatology (e.g., SEVIRI)	Decadal trend analysis and baseline mapping	(96, 250, 250)	48 MB (Float32, 1 channel)
Level-2 Derived Wind / Cloud Vectors	Regional synoptic weather monitoring	(24, 1000, 1000)	96 MB (Float32, 1 channel)

Compression Codecs

Firecube applies modern compression algorithms to balance high compression ratios with fast decompression speeds. The standard compressor used across Firecube Zarr products is Blosc, an open-source meta-compressor designed for fast scientific computing. Within Blosc, Firecube typically pairs the Zstandard (Zstd) or LZ4 algorithm with byte-shuffling or bit-shuffling filters.

The shuffle filter rearranges the physical bytes of the data array. Because multi-byte floating-point scientific numbers (such as 32-bit floats) often share identical exponent bits across spatially continuous fields, shuffling places identical bits adjacent to one another in memory. This step significantly improves the compression ratio achieved by Zstandard without requiring slower, more aggressive compression levels.

Ingestion and Write Strategies

Populating a cloud-native Zarr store from thousands of asynchronous batch jobs requires careful write coordination. Firecube uses two distinct write strategies depending on operational constraints: sequential batch appending and direct region-based writing.

Sequential Batch Appending

In sequential appending, Firecube receives a batch of granules covering a discrete time window, constructs an in-memory xarray.Dataset, and appends it to the existing Zarr store along the time dimension using the parameter append_dim="time". In this mode, the target Zarr store does not need to know its final temporal extent in advance.

However, pure sequential appending presents limitations in high-throughput production environments:

• The client must read the existing root metadata, modify the temporal dimension shape in .zarray, and write the updated JSON file back to object storage.
• If two distributed workers attempt to append batches simultaneously, a metadata write race condition occurs. One worker may overwrite the dimension updates of the other, resulting in orphaned data chunks that are physically present in the S3 bucket but unindexed by the dataset's metadata.
• It requires external locking mechanisms (such as Redis-based locks or Consul key-value locks) to guarantee atomic metadata updates, which introduces infrastructural complexity.

Predefined Region Writing

To support high-concurrency ingestion across parallel cloud instances, Firecube uses direct region-based writing via the region argument supported by zarr-python and xarray. This architecture bypasses metadata-locking issues entirely through a two-phase process:

1. Store Initialization (Skeleton Phase): An initialization worker defines the complete dataset structure in advance. It writes the global .zgroup, coordinate dimensions, and all variable .zarray and .zattrs metadata files to S3, establishing the final expected shape of the dataset (e.g., 10 years of observation, consisting of 350,400 time steps). No data chunks are created during this phase; the array exists as a metadata "skeleton" populated with logical NaN fill values.
2. Concurrent Region Ingestion: Distributed Firecube worker processes are assigned specific, non-overlapping coordinate regions. For example, Worker A is assigned time slice [0:96], while Worker B is assigned time slice [96:192]. Each worker writes its payload directly to the storage keys corresponding to its assigned slice.

Because every worker writes exclusively to unique object storage keys (such as radiance/0.0.0.0 vs radiance/96.0.0.0) and none of the workers modify the global .zarray metadata files, write contention is eliminated. This design enables linear scaling of ingestion throughput across distributed workers on the European Weather Cloud.

Chunk Boundary Alignment Constraints

Region writing requires that the temporal bounds of an ingestion batch align precisely with the underlying Zarr chunk boundaries. If a Zarr array is configured with a chunk size of 24 time steps along the time dimension (e.g., chunk_shape = (24, 500, 500)), any worker writing a region must write in multiples of 24 time steps.

If an ingestion batch containing only 12 time steps attempts to write into a chunk of size 24, it must first read the remaining 12 time steps from the existing chunk on S3, merge the new data into the local memory array, and write back the entire 24-step compressed chunk. This read-modify-write cycle reintroduces concurrency vulnerabilities. Firecube addresses this constraint by using its ChunkManager to enforce strict alignment between worker task schedules and physical chunk boundaries.

Analysis-Ready Data (ARD) Transformation

A primary goal of Firecube Zarr products is delivering Analysis-Ready Data (ARD). The Committee on Earth Observation Satellites (CEOS) defines Analysis-Ready Data as satellite data that have been processed to a minimum set of requirements and organized in a form that allows immediate analysis with minimal additional user effort.

Radiometric Normalisation

Raw satellite telemetry transmits observations as integer digital counts (typically 10-bit to 14-bit unsigned integers) to optimise downlink bandwidth. Traditionally, downstream users had to parse calibration coefficients—often variable per detector and subject to drift compensation updates—from secondary header tables to compute physical radiances. Firecube applies these radiometric calibration tables during the ingestion pipeline, writing out physical variables directly:

• Top-of-Atmosphere (TOA) Radiance: Calibrated spectral radiance expressed in standard units ($W \cdot m^{-2} \cdot sr^{-1} \cdot \mu m^{-1}$).
• Reflectance: For solar reflective bands, raw radiances are normalized by solar zenith angle and extraterrestrial solar irradiance, yielding dimensionless reflectance values scaled from 0.0 to 1.0.
• Brightness Temperature: For thermal infrared channels, radiances are converted to equivalent blackbody brightness temperatures in Kelvins ($K$) using the Planck function and instrument-specific spectral response band corrections.

Geospatial Standardization

Native satellite swaths have spatial coordinates that vary continuously across scan lines due to spacecraft ephemeris, attitude drift, and optical geometry. Firecube standardises these coordinate systems:

• For geostationary satellites, spatial arrays are mapped to fixed grid coordinates defined in meters or radians on the projection plane, specified via standard Climate and Forecast (CF) Metadata Conventions attributes (such as grid_mapping_name = "geostationary", perspective_point_height, semi_major_axis, and inverse_flattening).
• For re-gridded products, coordinates are mapped onto regular equirectangular grids indexed by unambiguous one-dimensional vectors of latitude and longitude.

Quality Flags and Cloud Masking

Satellite observation suites include automated quality assessments, such as surface contamination flags, sensor saturation warnings, lunar contamination indicators, and cloud detection products. In native granule archives, these flags are often bit-packed into unsigned byte arrays requiring custom bitwise operations to interpret. Firecube unrolls or standardizes these masks into dedicated, indexed quality arrays (such as data_quality_flag) and documents individual bit meanings in the array's .zattrs metadata using standard CF flag_masks and flag_meanings attributes.

User Access and Scientific Workflows

Because Firecube Zarr products adhere to the open Zarr specification and CF conventions, they integrate directly into the scientific Python ecosystem without requiring custom software development kits or bespoke decompression tools.

Software Ecosystem Integration

Users access Firecube Zarr products through a standard stack of open-source tools:

• s3fs: A Python interface to S3-compatible object stores built on aiobotocore, providing an abstract file-system interface for HTTP/S3 requests.
• zarr-python: The reference implementation of the Zarr storage specification, handling chunk decoding, decompressor pipeline execution, and array access.
• xarray: A Python library for labeled multidimensional arrays, providing coordinate-aware indexing, slicing, aggregation, and mathematical transforms.
• Dask: A flexible library for parallel and distributed computing in Python that integrates with xarray to execute out-of-core calculations across multi-node clusters.

Data Access Example

The following Python script illustrates how a researcher accesses a Firecube Zarr store on the European Weather Cloud, selects a regional spatial bounding box and a distinct temporal window, and computes a temporal mean without downloading the full archive.

import xarray as xr
import s3fs

# 1. Configure the S3 file system interface for the European Weather Cloud
s3 = s3fs.S3FileSystem(
    anon=False,
    key="YOUR_EWC_ACCESS_KEY",
    secret="YOUR_EWC_SECRET_KEY",
    client_kwargs={"endpoint_url": "https://object-store.os-api.ewc.int"}
)

# 2. Point to the S3 bucket hosting the Firecube Zarr store
zarr_store_url = "s3://eumetsat-firecube-products/msg-seviri-level1b.zarr"
store_map = s3fs.S3Map(root=zarr_store_url, s3=s3, check=False)

# 3. Open the dataset lazily using consolidated metadata
ds = xr.open_dataset(
    store_map, 
    engine="zarr", 
    consolidated=True,
    chunks={"time": 24, "y": 500, "x": 500}
)

# 4. Extract an area of interest over Europe across a one-month duration
subset = ds["radiance"].sel(
    channel="IR_10.8",
    time=slice("2023-07-01T00:00:00", "2023-07-31T23:45:00"),
    y=slice(4200000, 3000000),
    x=slice(-1500000, 1500000)
)

# 5. Compute the temporal mean top-of-atmosphere radiance
# Dask executes parallel HTTP range-requests only for the required chunks
mean_radiance = subset.mean(dim="time").compute()

print(mean_radiance)

When executing this code, xr.open_dataset reads only the consolidated metadata files (.zmetadata), requiring less than 100 milliseconds. The underlying science data is not read until the explicit .compute() method is executed. At that point, Dask calculates the exact spatial and temporal chunk indices intersecting the requested slices, generates an execution graph, and issues parallel HTTP range requests fetching only the required chunks.

Performance and Storage Benchmarks

The transition from traditional granule archives to Firecube Zarr products yields measurable improvements across data retrieval throughput, processing latency, and computational efficiency.

Access Latency Comparison

In benchmark evaluations measuring time-series extractions over a single geographical pixel across 10,000 continuous time steps (representing approximately 100 days of 15-minute observations):

Storage Pattern	Number of Network Operations	Data Transferred	Execution Time
Granular netCDF-4 files on POSIX filesystem	10,000 file-opens + seek calls	~800 GB (entire files parsed)	14 minutes, 20 seconds
Granular netCDF-4 files on S3 object storage	30,000+ HTTP GET requests (headers + blocks)	~800 GB	1 hour, 45 minutes
Firecube Zarr datacube on S3 (optimized chunks)	416 HTTP GET requests	~1.2 GB (only relevant chunks fetched)	8.4 seconds

The significant reduction in execution time for Zarr stores is driven by two factors: minimizing the total number of HTTP requests and avoiding the transmission of unneeded spatial pixels surrounding the target coordinate.

Storage and Object Count Management

While Zarr datacubes deliver substantial read-performance advantages, they introduce challenges for object storage infrastructure. File systems track files using directory trees, whereas object storage systems track objects in flat bucket namespaces backed by internal database tables. Storing an operational 10-year satellite record partitioned into 10-megabyte chunks can generate 50 million to 100 million distinct S3 objects.

Within Ceph storage clusters, handling hundreds of millions of small objects places heavy demands on Object Storage Daemon (OSD) rocksdb metadata engines. This can increase administrative overhead for bucket indexing, cluster re-balancing, and data recovery tasks. Consequently, configuring Firecube chunk sizes requires balancing read performance against cluster storage engineering limits.

Comparison with Alternative Cloud Formats

Several cloud-native data architectures exist alongside native Zarr products in Earth observation. Understanding their relative trade-offs illustrates Firecube's specific design choices.

Format / Approach	Storage Type	Strengths	Weaknesses
Firecube Zarr	Native Zarr store on S3	Optimal read throughput; unified metadata; natively optimized chunk shapes across all dimensions.	Requires duplicating or permanently converting legacy raw data archives; produces millions of objects.
Cloud-Optimized GeoTIFF (COG)	Standalone 2D TIFF files on S3	Extensively supported by GIS software (QGIS, GDAL, Mapbox); excellent for 2D spatial rendering.	Inefficient for higher-dimensional arrays (N > 2); lacks a native model for multi-channel, multi-temporal time series.
Virtual Datacubes (Kerchunk)	Reference JSON/Parquet mapping over netCDF/HDF	Zero-copy; references existing archival netCDF/HDF5 files in place without moving raw data bytes.	Read performance remains constrained by original, non-cloud-optimised physical chunk layouts within source files.
TileDB	Multi-dimensional array engine	Excellent support for sparse and dense arrays; integrated security access controls.	Less common in open-source meteorological workflows; requires proprietary or specialized C++ runtime bindings.

Current Limitations, Technical Debates, and Future Directions

While Firecube Zarr products are deployed operationally within the European Weather Cloud, several technical challenges remain active topics of engineering development.

Near-Real-Time Ingestion Latency

Operational weather forecasting demands Near-Real-Time (NRT) data availability: observational data must be ready for ingestion into Numerical Weather Prediction (NWP) models within 5 to 15 minutes of being measured at the satellite sensor. Firecube's batch ingestion paradigm, designed around chunk boundaries and region updates, creates a trade-off between chunk size and ingestion frequency.

If a product uses chunks containing 24 time steps (6 hours of data), Firecube cannot commit a finalized chunk until all 24 time steps have been acquired and written. Holding chunks open in a temporary state or writing unaligned sub-chunks introduces write overhead and consistency hazards. As a result, operational operational meteorology workflows often rely on standard granule streams for initial NRT data assimilation, backfilling the consolidated Firecube Zarr stores later for climate services and machine learning research.

The Zarr Specification Version 3 and Sharding

The development of the Zarr Version 3 specification directly addresses the object-count problem seen in large-scale Earth observation archives. A key feature of Zarr v3 is sharding.

Under a sharded storage model, the logical chunk structure used by analysis software is decoupled from the physical object structure on the storage tier. For example, an array may be logically chunked into small (1, 250, 250) grids to enable precise spatial and temporal subsetting. Instead of storing each chunk as a separate object, the sharding codec combines hundreds of logical chunks into a single large physical object called a shard (e.g., 50 megabytes in size). An internal byte-index at the end of each shard points to the byte offsets of the constituent chunks.

When clients query a sharded Zarr v3 store, libraries like zarr-python read the shard's byte index and issue an HTTP GET request with a Range: bytes=start-end header to retrieve only the required logical chunk. This architecture reduces the total number of objects in the S3 bucket by two to three orders of magnitude, lowering overhead on Ceph storage clusters while preserving fine-grained chunk access for analysis.

Federation Across European Initiatives

As the European Commission and EUMETSAT expand initiatives such as Destination Earth (DestinE) and the Copernicus Data Space Ecosystem (CDSE), data architectures are moving toward distributed access across distinct cloud environments. Establishing federated Firecube Zarr stores across multiple European data centers introduces data synchronization challenges. Future work focuses on using high-performance cataloguing systems—such as SpatioTemporal Asset Catalogs (STAC)—to expose Firecube Zarr stores alongside distributed compute engines without requiring centralized data replication.

See also

• Cloud-Optimised Geospatial Data
• European Organisation for the Exploitation of Meteorological Satellites
• Hierarchical Data Format
• Meteosat Second Generation
• Meteosat Third Generation
• Network Common Data Form
• Xarray
• Zarr (data format)