Cargando…
Perspectives on incorporating expert feedback into model updates
Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts’ values and goals. However, there has been insufficient consideration of how practitioners should translate domain expertise into ML updates. In this review, we consider how...
Autores principales: | , , , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
Elsevier
2023
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10382980/ https://www.ncbi.nlm.nih.gov/pubmed/37521050 http://dx.doi.org/10.1016/j.patter.2023.100780 |
_version_ | 1785080794317848576 |
---|---|
author | Chen, Valerie Bhatt, Umang Heidari, Hoda Weller, Adrian Talwalkar, Ameet |
author_facet | Chen, Valerie Bhatt, Umang Heidari, Hoda Weller, Adrian Talwalkar, Ameet |
author_sort | Chen, Valerie |
collection | PubMed |
description | Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts’ values and goals. However, there has been insufficient consideration of how practitioners should translate domain expertise into ML updates. In this review, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation or domain level and then convert this feedback into updates to the dataset, loss function, or parameter space. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy and highlight the insufficient consideration given to incorporating feedback from non-technical experts. We end with a set of open questions that naturally arise from our proposed taxonomy and subsequent survey. |
format | Online Article Text |
id | pubmed-10382980 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2023 |
publisher | Elsevier |
record_format | MEDLINE/PubMed |
spelling | pubmed-103829802023-07-30 Perspectives on incorporating expert feedback into model updates Chen, Valerie Bhatt, Umang Heidari, Hoda Weller, Adrian Talwalkar, Ameet Patterns (N Y) Review Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts’ values and goals. However, there has been insufficient consideration of how practitioners should translate domain expertise into ML updates. In this review, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation or domain level and then convert this feedback into updates to the dataset, loss function, or parameter space. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy and highlight the insufficient consideration given to incorporating feedback from non-technical experts. We end with a set of open questions that naturally arise from our proposed taxonomy and subsequent survey. Elsevier 2023-07-14 /pmc/articles/PMC10382980/ /pubmed/37521050 http://dx.doi.org/10.1016/j.patter.2023.100780 Text en © 2023 The Author(s) https://creativecommons.org/licenses/by/4.0/This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/). |
spellingShingle | Review Chen, Valerie Bhatt, Umang Heidari, Hoda Weller, Adrian Talwalkar, Ameet Perspectives on incorporating expert feedback into model updates |
title | Perspectives on incorporating expert feedback into model updates |
title_full | Perspectives on incorporating expert feedback into model updates |
title_fullStr | Perspectives on incorporating expert feedback into model updates |
title_full_unstemmed | Perspectives on incorporating expert feedback into model updates |
title_short | Perspectives on incorporating expert feedback into model updates |
title_sort | perspectives on incorporating expert feedback into model updates |
topic | Review |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10382980/ https://www.ncbi.nlm.nih.gov/pubmed/37521050 http://dx.doi.org/10.1016/j.patter.2023.100780 |
work_keys_str_mv | AT chenvalerie perspectivesonincorporatingexpertfeedbackintomodelupdates AT bhattumang perspectivesonincorporatingexpertfeedbackintomodelupdates AT heidarihoda perspectivesonincorporatingexpertfeedbackintomodelupdates AT welleradrian perspectivesonincorporatingexpertfeedbackintomodelupdates AT talwalkarameet perspectivesonincorporatingexpertfeedbackintomodelupdates |