Mutual stain conversion between Giemsa and Papanicolaou in cytological images using cycle generative adversarial network

OBJECTIVE: Papanicolaou and Giemsa stains used in cytology have different characteristics and complementary roles. In this study, we focused on cycle-consistent generative adversarial network (CycleGAN), which is an image translation technique using deep learning, and we conducted mutual stain conversion between Giemsa and Papanicolaou in cytological images using CycleGAN. METHODS: A total of 191 Giemsa-stained images and 209 Papanicolaou-stained images were collected from 63 patients with lung cancer. From those images, 67 images from nine cases were used for testing and the remaining images were used for training. For data augmentation, the number of training images was increased by rotation and inversion, and the images were pipelined to CycleGAN to train the mutual conversion process involving Giemsa- and Papanicolaou-stained images. Three pathologists and three cytotechnologists performed visual evaluations of the authenticity of cell nuclei, cytoplasm, and cell layouts of the test images translated using CycleGAN. RESULTS: As a result of converting Giemsa-stained images into Papanicolaou-stained images, the background red blood cell patterns present in Giemsa-stained images disappeared, and cell patterns that reproduced the shape and staining of the cell nuclei and cytoplasm peculiar to Papanicolaou staining were obtained. Regarding the reverse-translated results, nuclei became larger, and red blood cells that were not evident in Papanicolaou-stained images appeared. After visual evaluation, although actual images exhibited better results than converted images, the results were promising for various applications. DISCUSSION: The stain translation technique investigated in this paper can complement specimens under conditions where only single stained specimens are available; it also has potential applications in the massive training of artificial intelligence systems for cell classification, and can also be used for training cytotechnologist and pathologists.