Multi-source satellite imagery ARTificial INtelligence Analysis challenge

ongoing

The increasing amounts of satellite data, combined with the increased number of areas of interest to monitor or retrieve fresh intelligence from, pushes the need for reliable automation of the analysis and reporting process. Artificial Intelligence (AI), Machine Learning (ML) and Large Language Model (LLM) techniques are rapidly developing, also in the satellite imagery analysis process tasks but objective evaluation of the performance of these solutions is currently not feasible. To address this problem MARTINA organises a 4 year technical challenge. The total duration of the challenge will be four years with one campaign per year. It will provide a framework and extensive reference data set to assess and evaluate the accuracy and reliability of AI/ML based automated satellite imagery analysis solutions for different defence use cases offered by three participating consortia. There is an additional focus on combining multiple image data types with different characteristics like, electro optical and radar images, to maximise information extraction. MARTINA will develop the required infra structure and software platform, develop clear guidelines and procedures to establish the transparent and realistic datasets for defence specific use cases, collect and label/annotate the required imagery for the tests, establish clear and objective evaluation methods and perform the actual tests and evaluations. MARTINA will engage with EU stakeholders in order to provide representative use cases of interest for defence, and to contribute to collaboration and knowledge sharing at the EU level. The stakeholders will also be involved in the definition of the (annotated) datasets. The emphasis of the evaluation will be on the performance in terms of reliable correct detection, classification and characterization and less on the performance in throughput and speed. Computing requirements or scalability of the solutions is not specifically addressed in this challenge but solution suppliers will have to address these topics in commercial applications. Each campaign exists of the selection of the use cases, data requirement definition (shared with challenge participants), data collection, data annotation and labelling, sharing of training set to the participants, the actual test on the test data in a limited time period and in a controlled manner and the evaluation of the results from each of the participants. The first campaign will be a test run to check and fine tune the procedures, data formats and workflows between the organising consortium and the participating consortia to provide an agreed testing environment and procedures. The other three campaigns will focus on specific use cases with increasing complexity and integration of advanced reporting aspects. The User Board is a crucial element of MARTINA, consisting of representatives from various European Ministries of Defense (MoDs) and other relevant EU entities. Its responsibilities include providing strategic guidance during each campaign, ensuring that use cases and locations are relevant to real-world defense scenarios, and contributing to the evaluation criteria for the AI applications developed by the participants. They will also test the demonstrators built by the participants in a qualitative manner, with aim to provide the possibility to test it with their own data in their own secure environment. The involvement is expected to require a commitment of approximately 5-10 days per year. The active engagement of the user board is essential for delivering high-impact, actionable solutions for European defense operations.