Cargando…
The Problem of Limited Inter-rater Agreement in Modelling Music Similarity
One of the central goals of Music Information Retrieval (MIR) is the quantification of similarity between or within pieces of music. These quantitative relations should mirror the human perception of music similarity, which is however highly subjective with low inter-rater agreement. Unfortunately t...
Autores principales: | , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Routledge
2016
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5256035/ https://www.ncbi.nlm.nih.gov/pubmed/28190932 http://dx.doi.org/10.1080/09298215.2016.1200631 |
_version_ | 1782498636282724352 |
---|---|
author | Flexer, Arthur Grill, Thomas |
author_facet | Flexer, Arthur Grill, Thomas |
author_sort | Flexer, Arthur |
collection | PubMed |
description | One of the central goals of Music Information Retrieval (MIR) is the quantification of similarity between or within pieces of music. These quantitative relations should mirror the human perception of music similarity, which is however highly subjective with low inter-rater agreement. Unfortunately this principal problem has been given little attention in MIR so far. Since it is not meaningful to have computational models that go beyond the level of human agreement, these levels of inter-rater agreement present a natural upper bound for any algorithmic approach. We will illustrate this fundamental problem in the evaluation of MIR systems using results from two typical application scenarios: (i) modelling of music similarity between pieces of music; (ii) music structure analysis within pieces of music. For both applications, we derive upper bounds of performance which are due to the limited inter-rater agreement. We compare these upper bounds to the performance of state-of-the-art MIR systems and show how the upper bounds prevent further progress in developing better MIR systems. |
format | Online Article Text |
id | pubmed-5256035 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2016 |
publisher | Routledge |
record_format | MEDLINE/PubMed |
spelling | pubmed-52560352017-02-08 The Problem of Limited Inter-rater Agreement in Modelling Music Similarity Flexer, Arthur Grill, Thomas J New Music Res Original Articles One of the central goals of Music Information Retrieval (MIR) is the quantification of similarity between or within pieces of music. These quantitative relations should mirror the human perception of music similarity, which is however highly subjective with low inter-rater agreement. Unfortunately this principal problem has been given little attention in MIR so far. Since it is not meaningful to have computational models that go beyond the level of human agreement, these levels of inter-rater agreement present a natural upper bound for any algorithmic approach. We will illustrate this fundamental problem in the evaluation of MIR systems using results from two typical application scenarios: (i) modelling of music similarity between pieces of music; (ii) music structure analysis within pieces of music. For both applications, we derive upper bounds of performance which are due to the limited inter-rater agreement. We compare these upper bounds to the performance of state-of-the-art MIR systems and show how the upper bounds prevent further progress in developing better MIR systems. Routledge 2016-07-02 2016-07-05 /pmc/articles/PMC5256035/ /pubmed/28190932 http://dx.doi.org/10.1080/09298215.2016.1200631 Text en © 2016 The Author(s). Published by Informa UK Limited, trading as Taylor & Francis Group http://creativecommons.org/licenses/by/4.0/ This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. |
spellingShingle | Original Articles Flexer, Arthur Grill, Thomas The Problem of Limited Inter-rater Agreement in Modelling Music Similarity |
title | The Problem of Limited Inter-rater Agreement in Modelling Music Similarity |
title_full | The Problem of Limited Inter-rater Agreement in Modelling Music Similarity |
title_fullStr | The Problem of Limited Inter-rater Agreement in Modelling Music Similarity |
title_full_unstemmed | The Problem of Limited Inter-rater Agreement in Modelling Music Similarity |
title_short | The Problem of Limited Inter-rater Agreement in Modelling Music Similarity |
title_sort | problem of limited inter-rater agreement in modelling music similarity |
topic | Original Articles |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5256035/ https://www.ncbi.nlm.nih.gov/pubmed/28190932 http://dx.doi.org/10.1080/09298215.2016.1200631 |
work_keys_str_mv | AT flexerarthur theproblemoflimitedinterrateragreementinmodellingmusicsimilarity AT grillthomas theproblemoflimitedinterrateragreementinmodellingmusicsimilarity AT flexerarthur problemoflimitedinterrateragreementinmodellingmusicsimilarity AT grillthomas problemoflimitedinterrateragreementinmodellingmusicsimilarity |