Active particle feedback control with a single-shot detection convolutional neural network

The real-time detection of objects in optical microscopy allows their direct manipulation, which has recently become a new tool for the control, e.g., of active particles. For larger heterogeneous ensembles of particles, detection techniques are required that can localize and classify different objects with strongly inhomogeneous optical contrast at video rate, which is often difficult to achieve with conventional algorithmic approaches. We present a convolutional neural network single-shot detector which is suitable for real-time applications in optical microscopy. The network is capable of localizing and classifying multiple microscopic objects at up to 100 frames per second in images as large as [Formula: see text] pixels, even at very low signal-to-noise ratios. The detection scheme can be easily adapted and extended, e.g., to new particle classes and additional parameters as demonstrated for particle orientation. The developed framework is shown to control self-thermophoretic active particles in a heterogeneous ensemble selectively. Our approach will pave the way for new studies of collective behavior in active matter based on artificial interaction rules.