Automated detection of smiles as discrete episodes

BACKGROUND: Patients seeking restorative and orthodontic treatment expect an improvement in their smiles and oral health‐related quality of life. Nonetheless, the qualitative and quantitative characteristics of dynamic smiles are yet to be understood. OBJECTIVE: To develop, validate, and introduce open‐access software for automated analysis of smiles in terms of their frequency, genuineness, duration, and intensity. MATERIALS AND METHODS: A software script was developed using the Facial Action Coding System (FACS) and artificial intelligence to assess activations of (1) cheek raiser, a marker of smile genuineness; (2) lip corner puller, a marker of smile intensity; and (3) perioral lip muscles, a marker of lips apart. Thirty study participants were asked to view a series of amusing videos. A full‐face video was recorded using a webcam. The onset and cessation of smile episodes were identified by two examiners trained with FACS coding. A Receiver Operating Characteristic (ROC) curve was then used to assess detection accuracy and optimise thresholding. The videos of participants were then analysed off‐line to automatedly assess the features of smiles. RESULTS: The area under the ROC curve for smile detection was 0.94, with a sensitivity of 82.9% and a specificity of 89.7%. The software correctly identified 90.0% of smile episodes. While watching the amusing videos, study participants smiled 1.6 (±0.8) times per minute. CONCLUSIONS: Features of smiles such as frequency, duration, genuineness, and intensity can be automatedly assessed with an acceptable level of accuracy. The software can be used to investigate the impact of oral conditions and their rehabilitation on smiles.