Deep learning detection of nanoparticles and multiple object tracking of their dynamic evolution during in situ ETEM studies

In situ transmission electron microscopy (TEM) studies of dynamic events produce large quantities of data especially under the form of images. In the important case of heterogeneous catalysis, environmental TEM (ETEM) under gas and temperature allows to follow a large population of supported nanoparticles (NPs) evolving under reactive conditions. Interpreting properly large image sequences gives precious information on the catalytic properties of the active phase by identifying causes for its deactivation. To perform a quantitative, objective and robust treatment, we propose an automatic procedure to track nanoparticles observed in Scanning ETEM (STEM in ETEM). Our approach involves deep learning and computer vision developments in multiple object tracking. At first, a registration step corrects the image displacements and misalignment inherent to the in situ acquisition. Then, a deep learning approach detects the nanoparticles on all frames of video sequences. Finally, an iterative tracking algorithm reconstructs their trajectories. This treatment allows to deduce quantitative and statistical features about their evolution or motion, such as a Brownian behavior and merging or crossing events. We treat the case of in situ calcination of palladium (oxide) / delta-alumina, where the present approach allows a discussion of operating processes such as Ostwald ripening or NP aggregative coalescence.