ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS validation series

Evaluating, optimising and benchmarking of next generation sequencing (NGS) variant calling performance are essential requirements for clinical, commercial and academic NGS pipelines. Such assessments should be performed in a consistent, transparent and reproducible fashion, using independently, ort...

Descripción completa

Detalles Bibliográficos
Autores principales: Ruark, Elise, Holt, Esty, Renwick, Anthony, Münz, Márton, Wakeling, Matthew, Ellard, Sian, Mahamdallie, Shazia, Yost, Shawn, Rahman, Nazneen
Formato: Online Artículo Texto
Lenguaje:English
Publicado: F1000 Research Limited 2018
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6234721/
https://www.ncbi.nlm.nih.gov/pubmed/30483600
http://dx.doi.org/10.12688/wellcomeopenres.14754.2
_version_ 1783370761687269376
author Ruark, Elise
Holt, Esty
Renwick, Anthony
Münz, Márton
Wakeling, Matthew
Ellard, Sian
Mahamdallie, Shazia
Yost, Shawn
Rahman, Nazneen
author_facet Ruark, Elise
Holt, Esty
Renwick, Anthony
Münz, Márton
Wakeling, Matthew
Ellard, Sian
Mahamdallie, Shazia
Yost, Shawn
Rahman, Nazneen
author_sort Ruark, Elise
collection PubMed
description Evaluating, optimising and benchmarking of next generation sequencing (NGS) variant calling performance are essential requirements for clinical, commercial and academic NGS pipelines. Such assessments should be performed in a consistent, transparent and reproducible fashion, using independently, orthogonally generated data. Here we present ICR142 Benchmarker, a tool to generate outputs for assessing germline base substitution and indel calling performance using the ICR142 NGS validation series, a dataset of Illumina platform-based exome sequence data from 142 samples together with Sanger sequence data at 704 sites. ICR142 Benchmarker provides summary and detailed information on the sensitivity, specificity and false detection rates of variant callers. ICR142 Benchmarker also automatically generates a single page report highlighting key performance metrics and how performance compares to widely-used open-source tools. We used ICR142 Benchmarker with VCF files outputted by GATK, OpEx and DeepVariant to create a benchmark for variant calling performance. This evaluation revealed pipeline-specific differences and shared challenges in variant calling, for example in detecting indels in short repeating sequence motifs. We next used ICR142 Benchmarker to perform regression testing with DeepVariant versions 0.5.2 and 0.6.1. This showed that v0.6.1 improves variant calling performance, but there was evidence of minor changes in indel calling behaviour that may benefit from attention. The data also allowed us to evaluate filters to optimise DeepVariant calling, and we recommend using 30 as the QUAL threshold for base substitution calls when using DeepVariant v0.6.1. Finally, we used ICR142 Benchmarker with VCF files from two commercial variant calling providers to facilitate optimisation of their in-house pipelines and to provide transparent benchmarking of their performance. ICR142 Benchmarker consistently and transparently analyses variant calling performance based on the ICR142 NGS validation series, using the standard VCF input and outputting informative metrics to enable user understanding of pipeline performance. ICR142 Benchmarker is freely available at https://github.com/RahmanTeamDevelopment/ICR142_Benchmarker/releases.
format Online
Article
Text
id pubmed-6234721
institution National Center for Biotechnology Information
language English
publishDate 2018
publisher F1000 Research Limited
record_format MEDLINE/PubMed
spelling pubmed-62347212018-11-26 ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS validation series Ruark, Elise Holt, Esty Renwick, Anthony Münz, Márton Wakeling, Matthew Ellard, Sian Mahamdallie, Shazia Yost, Shawn Rahman, Nazneen Wellcome Open Res Software Tool Article Evaluating, optimising and benchmarking of next generation sequencing (NGS) variant calling performance are essential requirements for clinical, commercial and academic NGS pipelines. Such assessments should be performed in a consistent, transparent and reproducible fashion, using independently, orthogonally generated data. Here we present ICR142 Benchmarker, a tool to generate outputs for assessing germline base substitution and indel calling performance using the ICR142 NGS validation series, a dataset of Illumina platform-based exome sequence data from 142 samples together with Sanger sequence data at 704 sites. ICR142 Benchmarker provides summary and detailed information on the sensitivity, specificity and false detection rates of variant callers. ICR142 Benchmarker also automatically generates a single page report highlighting key performance metrics and how performance compares to widely-used open-source tools. We used ICR142 Benchmarker with VCF files outputted by GATK, OpEx and DeepVariant to create a benchmark for variant calling performance. This evaluation revealed pipeline-specific differences and shared challenges in variant calling, for example in detecting indels in short repeating sequence motifs. We next used ICR142 Benchmarker to perform regression testing with DeepVariant versions 0.5.2 and 0.6.1. This showed that v0.6.1 improves variant calling performance, but there was evidence of minor changes in indel calling behaviour that may benefit from attention. The data also allowed us to evaluate filters to optimise DeepVariant calling, and we recommend using 30 as the QUAL threshold for base substitution calls when using DeepVariant v0.6.1. Finally, we used ICR142 Benchmarker with VCF files from two commercial variant calling providers to facilitate optimisation of their in-house pipelines and to provide transparent benchmarking of their performance. ICR142 Benchmarker consistently and transparently analyses variant calling performance based on the ICR142 NGS validation series, using the standard VCF input and outputting informative metrics to enable user understanding of pipeline performance. ICR142 Benchmarker is freely available at https://github.com/RahmanTeamDevelopment/ICR142_Benchmarker/releases. F1000 Research Limited 2018-10-31 /pmc/articles/PMC6234721/ /pubmed/30483600 http://dx.doi.org/10.12688/wellcomeopenres.14754.2 Text en Copyright: © 2018 Ruark E et al. http://creativecommons.org/licenses/by/4.0/ This is an open access article distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
spellingShingle Software Tool Article
Ruark, Elise
Holt, Esty
Renwick, Anthony
Münz, Márton
Wakeling, Matthew
Ellard, Sian
Mahamdallie, Shazia
Yost, Shawn
Rahman, Nazneen
ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS validation series
title ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS validation series
title_full ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS validation series
title_fullStr ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS validation series
title_full_unstemmed ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS validation series
title_short ICR142 Benchmarker: evaluating, optimising and benchmarking variant calling performance using the ICR142 NGS validation series
title_sort icr142 benchmarker: evaluating, optimising and benchmarking variant calling performance using the icr142 ngs validation series
topic Software Tool Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6234721/
https://www.ncbi.nlm.nih.gov/pubmed/30483600
http://dx.doi.org/10.12688/wellcomeopenres.14754.2
work_keys_str_mv AT ruarkelise icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries
AT holtesty icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries
AT renwickanthony icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries
AT munzmarton icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries
AT wakelingmatthew icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries
AT ellardsian icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries
AT mahamdallieshazia icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries
AT yostshawn icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries
AT rahmannazneen icr142benchmarkerevaluatingoptimisingandbenchmarkingvariantcallingperformanceusingtheicr142ngsvalidationseries