Cargando…
On the number of trials needed to distinguish similar alternatives
A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smalle...
Autores principales: | , , |
---|---|
Formato: | Online Artículo Texto |
Lenguaje: | English |
Publicado: |
National Academy of Sciences
2022
|
Materias: | |
Acceso en línea: | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9351503/ https://www.ncbi.nlm.nih.gov/pubmed/35901210 http://dx.doi.org/10.1073/pnas.2202116119 |
_version_ | 1784762458030735360 |
---|---|
author | Chierichetti, Flavio Kumar, Ravi Tomkins, Andrew |
author_facet | Chierichetti, Flavio Kumar, Ravi Tomkins, Andrew |
author_sort | Chierichetti, Flavio |
collection | PubMed |
description | A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smaller improvement in quality from the items produced by the current system A. By formalizing this observation, we develop closed-form analytical expressions for the sample efficiency of a number of widely used families of slate-based comparison tests. In empirical trials, these theoretical sample complexity results are shown to be predictive of real-world testing efficiency outcomes. These findings offer opportunities for both more cost-effective testing and a better analytical understanding of the problem. |
format | Online Article Text |
id | pubmed-9351503 |
institution | National Center for Biotechnology Information |
language | English |
publishDate | 2022 |
publisher | National Academy of Sciences |
record_format | MEDLINE/PubMed |
spelling | pubmed-93515032023-01-28 On the number of trials needed to distinguish similar alternatives Chierichetti, Flavio Kumar, Ravi Tomkins, Andrew Proc Natl Acad Sci U S A Social Sciences A/B testing is widely used to tune search and recommendation algorithms, to compare product variants as efficiently and effectively as possible, and even to study animal behavior. With ongoing investment, due to diminishing returns, the items produced by the new alternative B show smaller and smaller improvement in quality from the items produced by the current system A. By formalizing this observation, we develop closed-form analytical expressions for the sample efficiency of a number of widely used families of slate-based comparison tests. In empirical trials, these theoretical sample complexity results are shown to be predictive of real-world testing efficiency outcomes. These findings offer opportunities for both more cost-effective testing and a better analytical understanding of the problem. National Academy of Sciences 2022-07-28 2022-08-02 /pmc/articles/PMC9351503/ /pubmed/35901210 http://dx.doi.org/10.1073/pnas.2202116119 Text en Copyright © 2022 the Author(s). Published by PNAS https://creativecommons.org/licenses/by-nc-nd/4.0/This article is distributed under Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND) (https://creativecommons.org/licenses/by-nc-nd/4.0/) . |
spellingShingle | Social Sciences Chierichetti, Flavio Kumar, Ravi Tomkins, Andrew On the number of trials needed to distinguish similar alternatives |
title | On the number of trials needed to distinguish similar alternatives |
title_full | On the number of trials needed to distinguish similar alternatives |
title_fullStr | On the number of trials needed to distinguish similar alternatives |
title_full_unstemmed | On the number of trials needed to distinguish similar alternatives |
title_short | On the number of trials needed to distinguish similar alternatives |
title_sort | on the number of trials needed to distinguish similar alternatives |
topic | Social Sciences |
url | https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9351503/ https://www.ncbi.nlm.nih.gov/pubmed/35901210 http://dx.doi.org/10.1073/pnas.2202116119 |
work_keys_str_mv | AT chierichettiflavio onthenumberoftrialsneededtodistinguishsimilaralternatives AT kumarravi onthenumberoftrialsneededtodistinguishsimilaralternatives AT tomkinsandrew onthenumberoftrialsneededtodistinguishsimilaralternatives |