Automated analysis of computerized morphological features of cell clusters associated with malignancy on bile duct brushing whole slide images

BACKGROUND: Bile duct brush specimens are difficult to interpret as they often present inflammatory and reactive backgrounds due to the local effects of stricture, atypical reactive changes, or previously installed stents, and often have low to intermediate cellularity. As a result, diagnosis of biliary adenocarcinomas is challenging and often results in large interobserver variability and low sensitivity OBJECTIVE: In this work, we used computational image analysis to evaluate the role of nuclear morphological and texture features of epithelial cell clusters to predict the presence of pancreatic and biliary tract adenocarcinoma on digitized brush cytology specimens. METHODS: Whole slide images from 124 patients, either diagnosed as benign or malignant based on clinicopathological correlation, were collected and randomly split into training (S (T), N = 58) and testing (S ( v ), N = 66) sets, with the exception of cases diagnosed as atypical on cytology were included in S ( v ). Nuclear boundaries on cell clusters extracted from each image were segmented via a watershed algorithm. A total of 536 quantitative morphometric features pertaining to nuclear shape, size, and aggregate cluster texture were extracted from within the cell clusters. The most predictive features from patients in S (T) were selected via rank‐sum, t‐test, and minimum redundancy maximum relevance (mRMR) schemes. The selected features were then used to train three machine‐learning classifiers. RESULTS: Malignant clusters tended to exhibit lower textural homogeneity within the nucleus, greater textural entropy around the nuclear membrane, and longer minor axis lengths. The sensitivity of cytology alone was 74% (without atypicals) and 46% (with atypicals). With machine diagnosis, the sensitivity improved to 68% from 46% when atypicals were included and treated as nonmalignant false negatives. The specificity of our model was 100% within the atypical category. CONCLUSION: We achieved an area under the receiver operating characteristic curve (AUC) of 0.79 on S ( v ), which included atypical cytological diagnosis.