Validation of an algorithm for automated classification of migraine and tension-type headache attacks in an electronic headache diary

BACKGROUND: This study evaluates the accuracy of an automated classification tool of single attacks of the two major primary headache disorders migraine and tension-type headache used in an electronic headache diary. METHODS: One hundred two randomly selected reported headache attacks from an electronic headache-diary of patients using the medical app M-sense were classified by both a neurologist with specialisation in headache medicine and an algorithm, constructed based on the ICHD-3 criteria for migraine and tension-type headache. The level of agreement between the headache specialist and the algorithm was compared by using a kappa statistic. Cases of disagreement were analysed in a disagreement validity assessment. RESULT: The neurologist and the algorithm classified migraines with aura (MA), migraines without aura (MO), tension-type headaches (TTH) and non-migraine or non-TTH events. Of the 102 headache reports, 86 cases were fully agreed on, and 16 cases not, making the level of agreement unweighted kappa 0.74 and representing a substantial level of agreement. Most cases of disagreement (12 out of 16) were due to inadvertent mistakes of the neurologist identified in the disagreement validity assessment. The second most common reason (3 out of 16) was insufficient information for classification by the neurologist. CONCLUSIONS: The substantial level of agreement indicates that the classification tool is a valuable instrument for automated evaluation of electronic headache diaries, which can thereby support the diagnostic and therapeutic clinical processes. Based on this study’s results, additional diagnostic functionalities of primary headache management apps can be implemented. Finally, future research can use this classification algorithm for large scale database analysis for epidemiological studies.