Cargando…

On the number of trials needed to distinguish similar alternatives

A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smalle...

Descripción completa

Detalles Bibliográficos
Autores principales: Chierichetti, Flavio, Kumar, Ravi, Tomkins, Andrew
Formato: Online Artículo Texto
Lenguaje:English
Publicado: National Academy of Sciences 2022
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9351503/
https://www.ncbi.nlm.nih.gov/pubmed/35901210
http://dx.doi.org/10.1073/pnas.2202116119
_version_ 1784762458030735360
author Chierichetti, Flavio
Kumar, Ravi
Tomkins, Andrew
author_facet Chierichetti, Flavio
Kumar, Ravi
Tomkins, Andrew
author_sort Chierichetti, Flavio
collection PubMed
description A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smaller improvement in quality from the items produced by the current system A. By formalizing this observation, we develop closed-form analytical expressions for the sample efficiency of a number of widely used families of slate-based comparison tests. In empirical trials, these theoretical sample complexity results are shown to be predictive of real-world testing efficiency outcomes. These findings offer opportunities for both more cost-effective testing and a better analytical understanding of the problem.
format Online
Article
Text
id pubmed-9351503
institution National Center for Biotechnology Information
language English
publishDate 2022
publisher National Academy of Sciences
record_format MEDLINE/PubMed
spelling pubmed-93515032023-01-28 On the number of trials needed to distinguish similar alternatives Chierichetti, Flavio Kumar, Ravi Tomkins, Andrew Proc Natl Acad Sci U S A Social Sciences A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smaller improvement in quality from the items produced by the current system A. By formalizing this observation, we develop closed-form analytical expressions for the sample efficiency of a number of widely used families of slate-based comparison tests. In empirical trials, these theoretical sample complexity results are shown to be predictive of real-world testing efficiency outcomes. These findings offer opportunities for both more cost-effective testing and a better analytical understanding of the problem. National Academy of Sciences 2022-07-28 2022-08-02 /pmc/articles/PMC9351503/ /pubmed/35901210 http://dx.doi.org/10.1073/pnas.2202116119 Text en Copyright © 2022 the Author(s). Published by PNAS https://creativecommons.org/licenses/by-nc-nd/4.0/This article is distributed under Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND) (https://creativecommons.org/licenses/by-nc-nd/4.0/) .
spellingShingle Social Sciences
Chierichetti, Flavio
Kumar, Ravi
Tomkins, Andrew
On the number of trials needed to distinguish similar alternatives
title On the number of trials needed to distinguish similar alternatives
title_full On the number of trials needed to distinguish similar alternatives
title_fullStr On the number of trials needed to distinguish similar alternatives
title_full_unstemmed On the number of trials needed to distinguish similar alternatives
title_short On the number of trials needed to distinguish similar alternatives
title_sort on the number of trials needed to distinguish similar alternatives
topic Social Sciences
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9351503/
https://www.ncbi.nlm.nih.gov/pubmed/35901210
http://dx.doi.org/10.1073/pnas.2202116119
work_keys_str_mv AT chierichettiflavio onthenumberoftrialsneededtodistinguishsimilaralternatives
AT kumarravi onthenumberoftrialsneededtodistinguishsimilaralternatives
AT tomkinsandrew onthenumberoftrialsneededtodistinguishsimilaralternatives