Earth Observation & Weather Data Federation with AI Embeddings

ongoing

The full potential of the Copernicus Programme unfolds when fused with additional geo-information such as weather models or GNSS measurements. However, no single platform can host all the hundreds of petabytes of geospatial data. Currently, service suppliers download data from different archives, and the sheer volume to be transferred render many applications economically not viable. With Embed2Scale we strive to overcome these limitations enabling efficient exchange of data through AI-based data compression. We will explore the training of deep neural networks on HPC systems with self-supervised learning to transform raw geo-information into embeddings with up to 1000-fold compression. The main innovations will enable i) decentralized applications through substantial reduction of “data gravity”, ii) the portability of geospatial analytics by significantly lowering computational demand, iii) minimizing data labeling by few-shot learning, and iv) the near-real-time similarity search at petabyte scale of Earth observation and weather/climate data archives. The objectives of Embed2Scale target i) the exploration of ground-breaking AI-compressors enabling data federation to proliferate a MLOps reference implementation for embeddings in data centers, ii) to demonstrate data federation on real-world use-cases for the Copernicus Programme, and iii) to enable the Earth observation community by open-sourcing and standardization. Within Embed2Scale, we will benchmark the use of embeddings in four applications: i) maritime awareness, ii) aboveground biomass estimation, iii) climate and air pollution prediction, and iv) crop stress & early yield detection. Overall, Embed2Scale will enable near-real time quantitative assessments of geo-information at continental scale - we respond to challenge 2 of the call: “new, enabling, scalable, operational solutions and technologies to improve capabilities of the Copernicus value chain and supporting infrastructure”.