IMPRIO IP Core: Image Prioritization for the OPIC Instrument in ESA’s Comet Interceptor Mission

Introduction

The primary outcome of scientific space missions is data — and the volume and scientific value of that data ultimately determine the mission’s return on investment. With flagship missions often exceeding €100 million, the natural expectation is to capture and transmit every possible measurement. In practice, however, missions are constrained by the limited communication bandwidth between spacecraft and Earth. As exploration extends into the outer Solar System, these constraints become even more critical, making efficient data handling strategies indispensable.

To overcome this challenge, missions typically apply two complementary approaches: data compression, to reduce the volume of information transmitted, and data prioritization, to ensure that the most critical observations are returned first. In practice, both methods are often combined to maximize mission return.

ESA’s Comet Interceptor mission, part of the agency’s F-class portfolio, is designed to encounter a pristine long-period comet entering the inner Solar System for the first time from the Oort Cloud. The mission comprises three spacecraft: a primary spacecraft (A) and two sub-spacecraft (B1 and B2). Onboard spacecraft B2, the University of Tartu’s Optical Periscopic Imager for Comets (OPIC) will capture images of the comet nucleus and its surrounding environment during the high-velocity flyby.

Figure from Snodgrass et al., 2023, licensed under CC BY 4.0.  

From Requirements to Constraints

To meet OPIC’s requirements, Bitlake developed the IMPRIO FPGA IP core to autonomously identify and prioritize ROIs in each image during the comet flyby. In practice, this means locating the comet nucleus within a captured frame and isolating it for transmission, since this portion of the image carries the highest scientific value.

Algorithm Selection

The IP core was required to process 2048 × 2048, 12-bit monochromatic images at a sustained throughput of at least 6 frames per second. This had to be achieved despite the unpredictable shape, size, and visual features of the comet nucleus — factors that complicate autonomous detection.

At first glance, such functionality might appear straightforward in today’s era of FPGA-accelerated machine learning. However, OPIC’s hardware imposed strict constraints. The IMPRIO IP core had to be integrated directly into the firmware of the OPIC camera head (3D Plus 3DCM734-1-SS), built on a robust and flash-based ProASIC3L FPGA. While highly reliable for spaceflight, this device is optimized for detector readout and robustness, not for image processing. It offers no dedicated DSP blocks, and much of its limited logic was already allocated to essential camera functions. These conditions transformed what might otherwise be a routine image-processing task into a highly constrained engineering challenge.

With the functionality defined and the hardware limitations established, the next step was to select an algorithm capable of delivering robust performance while remaining feasible on such a constrained platform. The solution chosen was a multiscale Laplacian of Gaussian (LoG) blob detection algorithm.

This approach provides several advantages:

  • FPGA suitability — based on convolution operations, which can be highly parallelized and optimized for a high throughput.
  • Scale adaptability — by analyzing images at multiple resolutions, the algorithm naturally compensates for the unknown and dynamically changing size of the comet nucleus during the flyby.

Robustness — the LoG method is inherently resilient to sensor noise, dead pixels, and illumination variability, making it particularly suited to the radiation environment of space.

Architecture and Pipeline

At the core of IMPRIO’s functionality lies the multiscale blob detection engine, supported by a lightweight cropping module that enables efficient readout of the selected ROI from off-chip memory. The architecture operates at four concurrent scales (full resolution, 1/2, 1/4, and 1/8), ensuring reliable detection across a wide range of apparent nucleus sizes.

The blob detection pipeline consists of three stages:

  1. Quantization and Downscaling — input frames are quantizied and downscaled to the required levels.
  2. Convolution with LoG Kernels — multi-scale filtering enhances blob-like structures corresponding to the nucleus.
  3. Analysis, ROI Selection, and Cropping — candidate regions are evaluated, the most relevant ROI is selected, and the corresponding segment is read out for prioritized transmission.

As illustrated in the example below, IMPRIO would have successfully isolated the nucleus in representative frames from ESA’s Rosetta mission flyby, demonstrating the practical outcome of the multiscale detection approach.

From an implementation standpoint, the architecture was optimized for the ProASIC3L FPGA by running the pipeline stages concurrently, enabling real-time parallelism while minimizing logic utilization. Convolution modules were carefully tailored to exploit the available resources without dedicated DSPs, and memory access was streamlined to support ROI cropping.

Verification and Implementation

Before being deployed in hardware, the functional model of IMPRIO was validated using several hundred synthetic images of comet nuclei generated by the OPIC team. Since real flyby images are limited, the synthetic dataset included challenging conditions such as noise, dead pixels, and variable lighting, ensuring the algorithm was tested against scenarios representative of the space environment.

Integration into the OPIC camera head represented the most demanding stage of the project. The ProASIC3L FPGA, while highly reliable, is not designed for image-processing applications. With much of its limited logic already occupied by essential camera functions — including detector readout, memory interfaces, and external communication — fitting IMPRIO required numerous iterations of timing closure, resource balancing, and architectural optimization. Ultimately, the core was successfully implemented and validated on the camera head engineering model, demonstrating that relatively high throughput image processing can be achieved even on platforms typically considered unsuitable for such tasks.

Performance and Results

After many iteration of optimization, the IMPRIO IP core was integrated into the OPIC firmware and subjected to laboratory-level tests involving image capture and prioritization. The IMPRIO IP core achieved ~7.1 frames per second performance on 2048 × 2048, 12-bit monochromatic images, surpassing the baseline requirement of 6 fps. This margin ensures that OPIC can reliably track and prioritize the comet nucleus during the brief and unrepeatable flyby, even under variable conditions of size, shape, and illumination.

Equally significant, this performance was delivered within the severe limitations of the ProASIC3L FPGA. Despite lacking DSP blocks and being heavily utilized by other camera functions, the device achieved timing closure with sufficient margin for stable, long-term operation in a radiation environment.

For the OPIC instrument, IMPRIO translates directly into maximized scientific return: the ability to capture and transmit the most critical data within tight bandwidth constraints.

This work is part of the ESA project “IP core for on-board blob detection and cropping in SSSB missions” (ESA AO/1-11736/23/NL/MH)