MTGProc and Satpy Satellite Data Processing Frameworks AI-pedia

Modern meteorological forecasting depends on rapid, automated processing of remote sensing data collected by Earth observation satellites. Instruments mounted on geostationary Earth orbit (GEO) platforms—orbiting approximately 35,786 kilometers above the equator—and low Earth orbit (LEO) platforms continuously measure electromagnetic radiation reflected and emitted by Earth's surface and atmosphere. Before these raw measurements can inform operational forecasters, numerical weather prediction models, or atmospheric researchers, they must undergo extensive calibration, geometric correction, projection remapping, and multispectral color compositing. Two major software frameworks employed within the European and international meteorological communities to accomplish these tasks are MTGProc and Satpy.

While both frameworks address the conversion of raw or semi-processed satellite radiances into scientifically meaningful visual products and numerical arrays, they represent fundamentally different engineering philosophies. MTGProc is a specialized, turnkey operational batch-processing application developed within operational circles—most notably associated with the Czech Hydrometeorological Institute (CHMI) and European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT) working groups. It is designed to rapidly ingest data from specific next-generation geostationary sensors and output standardized imagery for operational forecasting desks. In contrast, Satpy is a general-purpose, modular Python framework developed by the international Pytroll open-source community. Satpy provides an extensible application programming interface (API) capable of processing data from dozens of distinct satellite instruments across diverse orbital regimes, backed by distributed computation and lazy-evaluation paradigms.

The operational commissioning of next-generation geostationary platforms—specifically EUMETSAT's Meteosat Third Generation (MTG) and the United States National Oceanic and Atmospheric Administration's (NOAA) Geostationary Operational Environmental Satellite R-Series (GOES-R)—has increased data volumes by an order of magnitude compared to earlier systems. This surge in data volume has intensified the technical demands placed on software processing pipelines. Understanding the mechanical differences, computational models, resource requirements, and ideal deployment environments of MTGProc and Satpy is critical for satellite meteorologists, software engineers, and national weather services tasked with maintaining real-time Earth observation systems.

Background and Evolution of Meteorological Satellite Processing

Satellite meteorology relies on instruments known as radiometers, which measure electromagnetic radiation across defined wavelength bands (spectral channels) encompassing visible, near-infrared, and thermal infrared energy. The operational utility of these measurements depends on their temporal latency (the time elapsed between physical measurement and product delivery) and their visual or physical interpretability.

For two decades, European geostationary meteorology was anchored by Meteosat Second Generation (MSG) satellites, the first of which (Meteosat-8) launched in 2002. MSG carried the Spinning Enhanced Visible and InfraRed Imager (SEVIRI), a 12-channel instrument producing a full-disk scan of the Earth every 15 minutes, with a spatial resolution of 3 kilometers at the sub-satellite point (nadir) for standard channels and 1 kilometer for a broadband High Resolution Visible (HRV) channel. To serve operational weather forecasting, regional meteorological offices required automated software that could ingest SEVIRI data from the EUMETCast broadcast service, apply geographic projections, and render standard multi-channel visualizations within minutes of reception.

To satisfy this operational need, atmospheric scientist Martin Setvák and colleagues at CHMI, in close collaboration with EUMETSAT working groups, developed MSGProc. MSGProc was engineered as an automated, procedural processing tool focused on severe convective storm analysis, rapid nowcasting (weather forecasting for the zero-to-six-hour timeframe), and forecaster workstation integration. It established automated pipelines for complex visualization techniques, such as the "sandwich" product, which fuses high-resolution visible imagery with color-enhanced thermal infrared data.

The transition to Meteosat Third Generation (MTG), beginning with the launch of the MTG-I1 imaging satellite in December 2022, introduced the Flexible Combined Imager (FCI). The FCI expands spectral coverage to 16 channels, increases full-disk temporal resolution to 10 minutes (and 2.5 minutes over Europe in Rapid Scanning Service mode), and doubles the spatial resolution to 1 kilometer and 2 kilometer baselines, with selective high-resolution channels reaching 0.5 kilometers. Parallel advances occurred in the United States with NOAA's GOES-R series (GOES-16, 17, 18, and 19), which carry the 16-channel Advanced Baseline Imager (ABI). The resulting data volume per repeat cycle expanded from tens of megabytes to several gigabytes. This shift necessitated modernizing MSGProc into MTGProc, while simultaneously driving the development of distributed, modular processing frameworks like Satpy.

MTGProc: Dedicated Operational Batch Processing

MTGProc is an operational image generation pipeline designed specifically to process data from MTG-FCI and GOES-ABI. As the architectural heir to MSGProc, it prioritizes deterministic execution, minimal runtime complexity, and direct compatibility with meteorological display systems used on operational forecast floors.

Design Philosophy and Scope

MTGProc is not an open-ended software development library; rather, it is a specialized operational application. Its design philosophy centers on the premise that an operational forecast office requires a reliable, self-contained processing engine that takes newly arrived satellite files from an ingestion directory, performs a fixed series of calculations and projections, and writes display-ready raster files to an output directory without requiring bespoke software development from the end user.

Because its scope is tightly bound to geostationary nowcasting, MTGProc is pre-configured with the geometric, radiometric, and spectral parameters of the FCI and ABI instruments. Users interact with MTGProc primarily through configuration files (such as .ini or parameter text files) and command-line execution wrappers. The configuration defines:

• Input directories: Monitored locations where satellite data files (such as NetCDF-4 Level 1c tiles or full-disk files) arrive via dissemination streams.
• Map projections and regions of interest: Bounding coordinates, target pixel dimensions, and cartographic projections (such as Mercator, Lambert Conformal Conic, or Polar Stereographic).
• Product recipes: Explicit instructions for single-channel visualizations, mathematical channel differences, standard multispectral color composites (RGBs), and sandwich products.
• Output specifications: Destination directories, file naming conventions, graphic annotations (timestamps, color bars, geographic coastlines, political boundaries), and output file formats (typically GeoTIFF, uncompressed TIFF, or standard PNG).

Processing Pipeline

The internal pipeline of MTGProc follows a linear, procedural execution flow optimized for high-throughput batch execution:

1. Ingestion and Parsing: The program parses incoming data containers. For MTG-FCI, these are typically Level 1c files delivered in Network Common Data Form version 4 (NetCDF-4), partitioned into geographic chunks or complete repeat cycles.
2. Calibration and Conversion: Raw digital numbers or pre-calibrated radiances are converted into physically meaningful values: top-of-atmosphere (TOA) bidirectional reflectance factors for solar (visible and near-infrared) channels, and effective brightness temperatures (expressed in Kelvin) derived via the Planck function for thermal infrared channels.
3. Spatial Remapping: The geostationary viewing geometry (an oblique projection viewing the spherical Earth from a fixed point above the equator) is transformed to the user-specified cartographic grid. MTGProc uses internal interpolation algorithms optimized for speed across fixed target areas.
4. Product Generation: Multi-channel formulas are calculated pixel by pixel. Contrast adjustments, non-linear gamma curves, and static threshold cuts are applied according to standardized meteorological guidelines.
5. Rendering and Annotation: Color look-up tables (LUTs) are applied to single-channel and sandwich datasets. Geographic vector overlays (borders, coastlines, latitude/longitude grids) are burned into the image pixels if configured, or written as transparent layers.
6. Output Delivery: Final raster files are written to disk. Operational visualization workstations—such as NinJo, AWIPS-II, or Visual Weather—read these files immediately for display to operational forecasters.

The "Sandwich" Product Implementation

A signature capability inherited from MSGProc and refined in MTGProc is the high-fidelity rendering of convective storm "sandwich" products. Developed extensively by Martin Setvák to aid in identifying severe updrafts, cloud-top cooling rates, overshooting tops, and cold-U/cold-ring storm signatures, the sandwich product fuses two distinct physical regimes:

• Base Layer (Visible): A high-resolution visible channel (such as FCI 0.6 µm or GOES-ABI 0.64 µm) provides morphological context, displaying cloud-top texture, shadows, gravity waves, and structural collapsing at the sub-kilometer scale. This layer is rendered in grayscale.
• Top Layer (Thermal Infrared): The clean infrared window channel (such as FCI 10.5 µm or ABI 10.3 µm) measures cloud-top radiometric temperature, which correlates inversely with cloud-top height in the troposphere.

MTGProc combines these layers dynamically. Cloud tops warmer than a specified threshold (typically around -35°C to -40°C) remain fully transparent, leaving the underlying visible texture unobscured. For pixels colder than this threshold, MTGProc applies a non-linear color scale (ranging from yellow to red, purple, and white for the coldest convective cores, often dropping below -70°C) with variable opacity. The resulting blended image allows a forecaster to assess intense, deep convective cores and their micro-scale structural details simultaneously in a single product, eliminating the need to toggle between separate visible and infrared displays.

Satpy and the Pytroll Ecosystem: Modular Multi-Mission Architecture

Satpy represents a distinct software development paradigm. Rather than serving as an operational turnkey application, Satpy is a modular, object-oriented software library. It forms the core processing and product generation component of the Pytroll project, an international community-driven initiative started around 2009 by European national meteorological services—including the Swedish Meteorological and Hydrological Institute (SMHI), the Danish Meteorological Institute (DMI), and the Norwegian Meteorological Institute (MET Norway)—to create free, open-source Python utilities for Earth observation data processing.

Core Modules and Dependencies

Satpy operates within a broader ecosystem of interrelated Python packages, each managing a specific domain of remote sensing physics and spatial computation:

• pyresample: A geospatial library dedicated to remapping satellite swath data and geostationary projections onto arbitrary target grids using techniques such as nearest neighbor, bilinear interpolation, elliptical weighted averaging (EWA), and k-dimensional tree (k-d tree) search algorithms.
• trollimage: A dedicated image processing library providing color spaces, non-linear stretching curves, color pallets, and multi-channel compositors designed specifically for geophysical imagery.
• pyorbital: An orbital mechanics library that computes satellite ephemeris, orbital propagation, and solar geometry (such as solar zenith and azimuth angles) required for atmospheric corrections.
• trollsift: A string and path parsing utility that extracts metadata (timestamps, satellite identifiers, channel names) from filenames using standardized format strings.

Within Satpy itself, the computational architecture is built natively upon the modern scientific Python stack, specifically Xarray and Dask.

The Xarray and Dask Data Model

A key architectural feature of Satpy is its departure from loading raw data into contiguous in-memory arrays (such as standard NumPy arrays). Instead, satellite channels are loaded as xarray.DataArray objects, which wrap Dask computational arrays.

Dask enables two architectural capabilities:

1. Chunked Memory Footprints: Arrays are divided into smaller two-dimensional or three-dimensional rectangular blocks called "chunks." Operations (such as calibration or remapping) are executed on individual chunks sequentially or in parallel, preventing full-disk datasets from exceeding physical random-access memory (RAM).
2. Lazy Evaluation (Delayed Execution): When an operational script instructs Satpy to open a dataset, resample it, calculate an RGB difference, and adjust its gamma curve, Satpy does not execute these calculations immediately. Instead, it constructs a directed acyclic graph (DAG) representing the complete sequence of computational steps. The computation is executed only when the user explicitly triggers an output operation, such as saving a file to disk or converting the data into an in-memory image.

This design allows Dask to optimize the computation graph, prune unused steps, and execute tasks concurrently across multiple CPU cores or distributed cluster nodes.

The Scene Object Abstraction

In Satpy, all operations center on the Scene class. A Scene represents a collection of satellite datasets across a specific temporal window and spatial domain. The workflow within Satpy is fully programmatic, requiring the user to write Python code to orchestrate operations:

from satpy import Scene
import glob

# Identify input files (e.g., MTG-FCI Level 1c NetCDF files)
filenames = glob.glob("/data/satellite/FCI/*.nc")

# Initialize the Scene with a specific reader
scn = Scene(filenames=filenames, reader="fci_l1c_nc")

# Load specific spectral channels or composite recipes
scn.load(["vis_06", "ir_105", "natural_color", "severe_storms"])

# Resample the data to a predefined cartographic area
local_scene = scn.resample("CentralEurope_1km")

# Export standard meteorological imagery
local_scene.save_datasets(writer="simple_image",
                          datasets=["natural_color", "severe_storms"],
                          filename="{sensor}_{name}_{start_time:%Y%m%d_%H%M}.png")

Multi-Mission Sensor Readers and Extensibility

Satpy’s architecture uses dynamic plugin registries for readers, modifiers, compositors, and writers. Satpy contains over 50 built-in reader plugins, allowing it to process data from diverse satellite platforms using a unified API:

• Geostationary platforms: MTG-FCI, MSG-SEVIRI, GOES-R ABI, Himawari AHI, Elektro-L, and Fengyun-4 AGRI.
• Low Earth Orbit platforms: Suomi-NPP/JPSS VIIRS, Terra/Aqua MODIS, MetOp AVHRR/IASI, Sentinel-2 MSI, Sentinel-3 OLCI/SLSTR, and NOAA POES.
• Ancillary data formats: NWP model outputs (ECMWF, GFS), digital elevation models (DEM), and radar composites.

Users can add new instruments or custom proprietary data formats by writing a Python reader class that implements standard interfaces for time parsing, coordinate generation, and array loading. The remainder of Satpy's ecosystem—compositing algorithms, projection transformations, and output writers—remains fully compatible with the new data source without modification.

Architectural and Performance Comparison

Although MTGProc and Satpy can generate visually identical meteorological products, their architectural divergence leads to trade-offs in computational performance, resource footprints, ease of deployment, and operational flexibility.

Functional Dimension	MTGProc	Satpy (Pytroll)
Core Architecture	Dedicated, turnkey batch-processing executable wrapper.	Modular, object-oriented scientific Python library.
Execution Interface	Static configuration files (.ini) and command-line execution.	Programmatic Python API, embedded in scripts or web services.
Supported Sensors	Tailored specifically for MTG-FCI and GOES-ABI.	Multi-mission: >50 sensors across GEO, LEO, and airborne platforms.
Memory Paradigm	Procedural, stream/tile-optimized contiguous memory allocation.	Lazy evaluation, chunked memory grids powered by Dask and Xarray.
Resampling Engine	Hardcoded, optimized geographic coordinate transformations.	pyresample engine (k-d tree, bilinear, EWA, nearest neighbor).
Composite Logic	Pre-defined mathematical formulas and hardcoded shaders/LUTs.	Declarative YAML configuration files coupled to extensible Python classes.
Horizontal Scaling	Achieved via external batch job schedulers (e.g., Slurm, PBS, systemd).	Native distributed scaling via dask.distributed clusters.
Workstation Integration	Turnkey generation of standard formats (NinJo, AWIPS, GeoTIFF, PNG).	Customizable output writers (NetCDF/CF, GeoTIFF, Zarr, PNG, AWIPS).
Primary User Base	Operational weather forecasting offices, nowcasting desks.	Remote sensing researchers, data engineers, operational agencies.

Memory Footprint and Computational Scaling

The processing of MTG-FCI data presents a notable computational challenge. A single Level 1c full-disk repeat cycle contains 16 spectral channels, several of which feature 0.5-kilometer spatial grids resulting in arrays exceeding 22,000 × 22,000 pixels per channel. Uncompressed, a single full disk across all channels represents tens of gigabytes of raw floating-point data.

MTGProc addresses this volume by processing data in fixed spatial chunks or localized geographic tiles. Because the application knows the exact array dimensions and target projection parameters ahead of time, it allocates fixed memory buffers and discards processed tiles immediately after rasterization. This deterministic approach ensures a predictable, bounded memory footprint, allowing MTGProc to run reliably on dedicated ingestion servers with modest physical memory allocations.

Satpy’s memory management is mediated by Dask. While this provides flexibility, it introduces operational sensitivity to chunk sizing. If chunk sizes are configured too small, the overhead of managing millions of tiny graph tasks degrades execution speed and consumes memory managing the Dask graph. If chunk sizes are too large (for example, attempting to process an entire high-resolution full-disk channel as a single chunk), the host system may encounter out-of-memory (OOM) errors and crash. To maintain operational stability with MTG data in Satpy, systems engineers must tune Dask chunking parameters (such as setting chunks to 1000 × 1000 or 2000 × 2000 float32 elements) and balance worker thread counts against physical memory availability.

However, Satpy's architectural complexity provides superior horizontal scalability. By pairing Satpy with dask.distributed, an organization can distribute the processing of a single MTG-FCI full disk across a compute cluster containing dozens of physical nodes. One set of nodes can process thermal calibration, another can calculate Rayleigh scattering corrections on visible channels, and a third can execute coordinate remapping simultaneously, completing large batch transformations faster than a single-server process.

Customizability and Algorithmic Prototyping

Satpy provides distinct advantages in research, prototyping, and non-standard workflow development. In Satpy, composite recipes are decoupled from executable code and stored in structured YAML configuration files. For example, creating a custom multispectral composite combining novel FCI channel ratios requires only defining a new entry in a local composites.yaml file:

composites:
  custom_convective_index:
    compositor: !!python/name:satpy.composites.GenericCompositor
    prerequisites:
      - name: ir_105
        modifiers: [sunzen_corr]
      - name: ir_038
      - name: ir_133
    standard_name: custom_convective_index

Because Satpy exposes the underlying data as Xarray structures, researchers can intercept intermediate data arrays, execute complex statistical or machine learning operations (such as running inference via PyTorch or TensorFlow on satellite feature stacks), and feed the results back into Satpy's output pipeline.

In MTGProc, algorithmic alterations are constrained by the capabilities compiled into the application or exposed via its configuration syntax. Modifying core algorithms or introducing experimental non-linear compositing math outside the scope of its operational recipes requires direct modifications to the application's source code.

Meteorological Compositing Science and Standards

Both MTGProc and Satpy implement the multispectral compositing guidelines established by the World Meteorological Organization (WMO) and EUMETSAT. These standards convert complex multispectral information into standardized RGB (Red-Green-Blue) products, allowing forecasters to interpret atmospheric processes quickly without evaluating 16 individual spectral channels separately.

Physical Foundations of Standard RGBs

RGB imagery assigns individual spectral channels, or mathematical differences between channels, to the primary red, green, and blue display color guns. Channel differences isolate specific physical properties of clouds and the atmosphere:

• Particle Phase and Size: The difference between the 3.8 µm (or 3.9 µm) shortwave infrared channel and the 10.5 µm (or 10.8 µm) thermal infrared channel isolates cloud phase (water vs. ice) and droplet size. In daylight, 3.8 µm radiance includes both reflected solar energy and emitted thermal energy. Water droplets strongly reflect 3.8 µm radiation, whereas ice crystals absorb it. Subtracting thermal emission reveals the reflected component, distinguishing small liquid water droplets from large ice particles.
• Optical Thickness: Solar reflectance channels (such as 0.6 µm and 0.8 µm) correlate directly with cloud optical depth. Thicker clouds scatter more solar radiation and appear bright; thinner clouds allow radiation to pass through and appear dark.
• Atmospheric Moisture and Tropopause Dynamics: Water vapor channels (such as 6.2 µm and 7.3 µm) absorb radiation emitted from the lower troposphere. Subtracting a lower-peaking water vapor channel from an upper-peaking channel measures mid-to-upper tropospheric moisture gradients, highlighting dry intrusions, jet streaks, and potential vorticity anomalies associated with cyclogenesis.

Key Operational RGB Recipes

MTGProc and Satpy are both routinely configured to generate standard WMO RGB products. The physics of these products remain identical across both platforms, though their internal rendering paths differ:

• Natural Color RGB: Combines 1.6 µm (Red), 0.8 µm (Green), and 0.6 µm (Blue). Because ice strongly absorbs 1.6 µm radiation while liquid water reflects it, ice clouds appear distinctly cyan, liquid water clouds appear white, and ice-free land surfaces appear green.
• Airmass RGB: Assigns the difference between water vapor channels (6.2 µm – 7.3 µm) to Red, the difference between ozone absorption and infrared window (9.7 µm – 10.8 µm) to Green, and the upper-level water vapor channel (6.2 µm) to Blue. This composite identifies air mass boundaries, stratospheric intrusions, and jet stream locations.
• Severe Convective Storms RGB: Uses channel differences to highlight strong updrafts and microphysical glaciation:
  • Red: Water vapor difference (6.2 µm – 10.8 µm), highlighting cloud tops that have penetrated into the dry stratosphere.
  • Green: Microphysical difference (3.9 µm – 10.8 µm), identifying small ice particles associated with severe, rapidly ascending updrafts.
  • Blue: Optical thickness difference (1.6 µm – 0.6 µm), separating thick convective cores from surrounding terrain.
  In this composite, intense convective storms with small ice crystals appear bright yellow or orange, providing an immediate visual warning of severe weather.
• True Color Imagery: Simulates human visual perception by combining red, green, and blue visible light. Because geostationary satellites historically lacked a native green channel (a limitation shared by GOES-ABI and MTG-FCI, which possess blue and red channels but lack an explicit green band around 0.55 µm), both MTGProc and Satpy implement synthetic green algorithms. A synthetic green channel is derived by calculating a weighted combination of the blue band, the red band, and the 0.86 µm near-infrared "vegetation" band. Furthermore, both tools apply atmospheric corrections to account for Rayleigh scattering, which scatters shorter blue wavelengths and causes uncorrected images to appear hazy and washed out.

Rendering Mechanics

In MTGProc, standard RGBs are produced via static, highly optimized mathematical transformations mapped to fixed internal color spaces. This hardcoded approach maximizes operational execution speed and guarantees strict numerical reproducibility across consecutive operational cycles.

In Satpy, RGB rendering is managed by specialized compositor classes in trollimage and satpy.composites. Satpy supports dynamic features such as:

• Day/Night Blending: Composites like DayNightCompositor dynamically merge solar-dependent daytime RGBs (such as True Color or Natural Color) with nighttime infrared composites along the solar terminator, adjusting pixel weights based on the solar zenith angle calculated by pyorbital.
• Dynamic Contrast Stretching: Satpy supports histogram-based equalization, adaptive gamma adjustments, and percentile-based stretches computed across the active spatial domain, rather than relying exclusively on fixed static cutoffs.

Operational Deployment and System Integration

The operational value of satellite processing software is measured by its reliability and latency within real-time forecasting pipelines. When severe weather threatens life and property, national weather services require that satellite products reach forecasters within two to three minutes of scan completion.

The Dissemination and Reception Context

Operational deployment of MTGProc and Satpy typically occurs within a structured data ingestion pipeline:

1. Raw Telemetry Acquisition: Ground stations acquire raw microwave downlinks from the satellite. Central processing facilities (such as EUMETSAT headquarters in Darmstadt, Germany, or NOAA facilities in Maryland, USA) calibrate and geometrically correct the data into Level 1c formats.
2. Broadcast and Dissemination: Processed data streams are compressed and broadcast to users. In Europe, this occurs primarily through EUMETCast, a high-throughput digital video broadcasting system (DVB-S2) that uplinks data to telecommunications satellites for reception by local meteorological offices via satellite dish systems. Alternatively, data is delivered via high-speed terrestrial networks and cloud object stores (such as AWS S3 or EUMETSAT Data Tailor services).
3. Local Ingestion and Execution: A local processing server receives the incoming chunks. This is where MTGProc or Satpy executes.

MTGProc in the Operational Nowcasting Office

In an operational nowcasting desk, system simplicity and predictability are paramount. MTGProc is typically deployed on a dedicated server directly connected to the EUMETCast reception storage pool. File system monitoring daemons (such as inotify on Linux systems) detect the arrival of complete repeat cycles or individual tile segments.

MTGProc is invoked directly with arguments pointing to the target configuration file. It reads the files, maps them to the local operational forecast sectors (e.g., a national territory or a regional radar-coverage bounding box), generates the pre-configured RGBs and sandwich images, and writes them directly into shared file systems accessible by meteorological workstations. Because MTGProc operates as a self-contained processing utility, it has few runtime dependencies. It does not require complex Python virtual environments, managing C-library bindings (such as GEOS or PROJ version mismatches), or cluster orchestration systems. If the process encounters a corrupted data segment, it discards the packet, logs an error, and continues processing the next incoming dataset, preventing pipeline halts.

Satpy in Cloud, Multi-Sensor, and Service-Oriented Architectures

Satpy is well suited for modern service-oriented, cloud-native, and multi-mission architectures. As national weather services migrate infrastructure to private or public cloud environments (such as OpenStack, Amazon Web Services, or European Open Science Cloud platforms), processing pipelines are increasingly containerized using Docker and orchestrated using Kubernetes.

In these architectures, Satpy serves as the engine within microservices:

• Asynchronous Message Queues: An event-driven architecture using message brokers (such as RabbitMQ, Apache Kafka, or Pytroll's posttroll system) passes JSON-formatted notification events when new satellite granules arrive in object storage.
• Dynamic Worker Scaling: Satpy workers encapsulated in lightweight container images pull jobs from the queue, instantiate Scene objects, download only the required byte ranges from cloud object storage (using Cloud-Optimized GeoTIFFs or Zarr data stores), execute processing graphs, and push rendered products to cloud-based tile servers or web map services (WMS).
• Multi-Sensor Fusion: Satpy allows workflows that fuse data from disparate platforms. A forecaster monitoring a developing cyclonic storm can execute a single Satpy pipeline that remaps MTG-FCI geostationary imagery, MetOp-C AVHRR polar passes, and Sentinel-3 altimetry products onto an identical map projection, exporting unified multi-sensor diagnostic stacks.

Current State and Technical Challenges

The operational rollout of Meteosat Third Generation has introduced new technical challenges that continue to shape the development of both MTGProc and Satpy.

Handling the MTG Lightning Imager (LI)

A major advancement of the MTG platform is the Lightning Imager (LI), an optical instrument that continuously detects total lightning (both cloud-to-cloud and cloud-to-ground flashes) across the Earth disk. Unlike the FCI, which produces continuous two-dimensional raster grids at regular time intervals, the LI generates non-gridded, sparse, event-based point data. Each lightning event is recorded with precise geographic coordinates, a microsecond timestamp, and measured optical radiance.

Visualizing LI data requires accumulating point events over fixed time windows (e.g., 2.5, 5, or 10 minutes), clustering events into flashes and groups, rasterizing these clusters onto the FCI spatial projection grid, and generating dynamic alpha-blended overlays. The community around MTGProc has focused on integrating these point-to-raster transforms directly into operational batch rendering, ensuring forecasters see lightning flash density contours superimposed directly over visible and infrared sandwich products. Simultaneously, Satpy developers have expanded its ingestion architecture to handle lightning data containers natively, allowing LI accumulations to be treated as dynamic datasets that integrate into standard composite graphs alongside spectral bands.

Format Complexities and Data Fragmentation

MTG Level 1c data is distributed in complex NetCDF-4 containers that employ internal compression (such as blosc, zlib, or szip) and data chunking. Reading these files efficiently requires careful optimization of the underlying HDF5 and NetCDF C-libraries. Inefficient I/O access patterns can cause disk thrashing, where processing engines spend more time seeking byte offsets within compressed NetCDF containers than executing radiometric math.

Both MTGProc and Satpy have required targeted engineering to handle this data flow:

• MTGProc optimizes data reading through direct, sequential access paths designed specifically around EUMETSAT's standardized chunk dissemination patterns.
• Satpy relies on low-level Python HDF5 wrappers (such as h5py or netCDF4-python) combined with Dask's lazy reader mechanics. Significant development effort within the Pytroll community has focused on optimizing file readers to avoid reading unneeded metadata or channels, minimizing network latency when streaming data from remote cloud object storage.

Integration with Artificial Intelligence and Nowcasting Models

Operational meteorology is increasingly adopting artificial intelligence (AI) and machine learning (ML) architectures for automated cloud classification, convective initiation prediction, and precipitation estimation. These neural network architectures require input data formatted as multi-dimensional tensor arrays rather than pre-rendered image files.

This paradigm shift highlights the complementary nature of both software frameworks. MTGProc remains highly effective for human-in-the-loop operational forecasting, reliably producing the annotated visual imagery required by operational meteorologists making time-critical warning decisions. Conversely, Satpy’s native integration with Xarray and Dask makes it well suited for serving data directly into automated AI pipelines. Because Xarray datasets can be converted directly into NumPy arrays, Apache Arrow tables, or PyTorch tensors without writing intermediate raster files to disk, Satpy functions as a preprocessing front-end for deep learning systems that consume real-time satellite data to predict severe convective storms.

Summary

MTGProc and Satpy represent two distinct, successful approaches to processing meteorological satellite data. MTGProc is a focused, operational application tailored for rapid, predictable batch generation of display-ready imagery from MTG-FCI and GOES-ABI instruments, requiring minimal operational infrastructure overhead. Satpy is a flexible, multi-sensor scientific library designed for cross-platform remote sensing, complex dynamic compositing, distributed scaling, and research-to-operations pipelines. Modern national meteorological services often deploy both: using MTGProc to ensure reliable, high-speed visualization on the operational forecast floor, while deploying Satpy across broader data engineering, multi-mission analysis, and automated algorithmic research systems.

See also

• Advanced Baseline Imager (ABI): The primary multi-channel imaging radiometer deployed on NOAA's GOES-R series satellites.
• Dask: A flexible open-source library for parallel and distributed computing in Python, utilized by Satpy for out-of-core computation.
• EUMETCast: EUMETSAT's primary satellite- and terrestrial-based multicast dissemination system for operational environmental data.
• European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT): The intergovernmental agency responsible for the operation of European meteorological satellite systems.
• Flexible Combined Imager (FCI): The state-of-the-art 16-channel imaging radiometer carried on Meteosat Third Generation imaging satellites.
• Meteosat Second Generation (MSG): The predecessor series of European geostationary meteorological satellites carrying the SEVIRI instrument.
• Meteosat Third Generation (MTG): The current operational generation of European geostationary meteorological satellites.
• Nowcasting: Short-range weather forecasting focused on the zero-to-six-hour timeframe, heavily reliant on real-time satellite imagery and radar.
• Pytroll: The international community and open-source software collective developing Python libraries for Earth observation data processing.
• Spinning Enhanced Visible and InfraRed Imager (SEVIRI): The 12-channel radiometer that served as the primary payload for Meteosat Second Generation.
• Xarray: An open-source Python library that introduces labeled dimensions, coordinates, and attributes to multi-dimensional raw arrays.