Towards automatic recognition of pure and mixed stones using intra‐operative endoscopic digital images

OBJECTIVE: To assess automatic computer‐aided in situ recognition of the morphological features of pure and mixed urinary stones using intra‐operative digital endoscopic images acquired in a clinical setting. MATERIALS AND METHODS: In this single‐centre study, a urologist with 20 years' experience intra‐operatively and prospectively examined the surface and section of all kidney stones encountered. Calcium oxalate monohydrate (COM) or Ia, calcium oxalate dihydrate (COD) or IIb, and uric acid (UA) or IIIb morphological criteria were collected and classified to generate annotated datasets. A deep convolutional neural network (CNN) was trained to predict the composition of both pure and mixed stones. To explain the predictions of the deep neural network model, coarse localization heat‐maps were plotted to pinpoint key areas identified by the network. RESULTS: This study included 347 and 236 observations of stone surface and stone section, respectively; approximately 80% of all stones exhibited only one morphological type and approximately 20% displayed two. A highest sensitivity of 98% was obtained for the type ‘pure IIIb/UA’ using surface images. The most frequently encountered morphology was that of the type ‘pure Ia/COM’; it was correctly predicted in 91% and 94% of cases using surface and section images, respectively. Of the mixed type ‘Ia/COM + IIb/COD’, Ia/COM was predicted in 84% of cases using surface images, IIb/COD in 70% of cases, and both in 65% of cases. With regard to mixed Ia/COM + IIIb/UA stones, Ia/COM was predicted in 91% of cases using section images, IIIb/UA in 69% of cases, and both in 74% of cases. CONCLUSIONS: This preliminary study demonstrates that deep CNNs are a promising method by which to identify kidney stone composition from endoscopic images acquired intra‐operatively. Both pure and mixed stone composition could be discriminated. Collected in a clinical setting, surface and section images analysed by a deep CNN provide valuable information about stone morphology for computer‐aided diagnosis.