Cargando…

Design considerations for workflow management systems use in production genomics research and the clinic

The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are...

Descripción completa

Detalles Bibliográficos
Autores principales: Ahmed, Azza E., Allen, Joshua M., Bhat, Tajesvi, Burra, Prakruthi, Fliege, Christina E., Hart, Steven N., Heldenbrand, Jacob R., Hudson, Matthew E., Istanto, Dave Deandre, Kalmbach, Michael T., Kapraun, Gregory D., Kendig, Katherine I., Kendzior, Matthew Charles, Klee, Eric W., Mattson, Nate, Ross, Christian A., Sharif, Sami M., Venkatakrishnan, Ramshankar, Fadlelmola, Faisal M., Mainzer, Liudmila S.
Formato: Online Artículo Texto
Lenguaje:English
Publicado: Nature Publishing Group UK 2021
Materias:
Acceso en línea:https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8569008/
https://www.ncbi.nlm.nih.gov/pubmed/34737383
http://dx.doi.org/10.1038/s41598-021-99288-8
_version_ 1784594556105261056
author Ahmed, Azza E.
Allen, Joshua M.
Bhat, Tajesvi
Burra, Prakruthi
Fliege, Christina E.
Hart, Steven N.
Heldenbrand, Jacob R.
Hudson, Matthew E.
Istanto, Dave Deandre
Kalmbach, Michael T.
Kapraun, Gregory D.
Kendig, Katherine I.
Kendzior, Matthew Charles
Klee, Eric W.
Mattson, Nate
Ross, Christian A.
Sharif, Sami M.
Venkatakrishnan, Ramshankar
Fadlelmola, Faisal M.
Mainzer, Liudmila S.
author_facet Ahmed, Azza E.
Allen, Joshua M.
Bhat, Tajesvi
Burra, Prakruthi
Fliege, Christina E.
Hart, Steven N.
Heldenbrand, Jacob R.
Hudson, Matthew E.
Istanto, Dave Deandre
Kalmbach, Michael T.
Kapraun, Gregory D.
Kendig, Katherine I.
Kendzior, Matthew Charles
Klee, Eric W.
Mattson, Nate
Ross, Christian A.
Sharif, Sami M.
Venkatakrishnan, Ramshankar
Fadlelmola, Faisal M.
Mainzer, Liudmila S.
author_sort Ahmed, Azza E.
collection PubMed
description The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are the software components employed to fill this gap. This work provides an approach and systematic evaluation of key features of popular bioinformatics WfMSs in use today: Nextflow, CWL, and WDL and some of their executors, along with Swift/T, a workflow manager commonly used in high-scale physics applications. We employed two use cases: a variant-calling genomic pipeline and a scalability-testing framework, where both were run locally, on an HPC cluster, and in the cloud. This allowed for evaluation of those four WfMSs in terms of language expressiveness, modularity, scalability, robustness, reproducibility, interoperability, ease of development, along with adoption and usage in research labs and healthcare settings. This article is trying to answer, which WfMS should be chosen for a given bioinformatics application regardless of analysis type?. The choice of a given WfMS is a function of both its intrinsic language and engine features. Within bioinformatics, where analysts are a mix of dry and wet lab scientists, the choice is also governed by collaborations and adoption within large consortia and technical support provided by the WfMS team/community. As the community and its needs continue to evolve along with computational infrastructure, WfMSs will also evolve, especially those with permissive licenses that allow commercial use. In much the same way as the dataflow paradigm and containerization are now well understood to be very useful in bioinformatics applications, we will continue to see innovations of tools and utilities for other purposes, like big data technologies, interoperability, and provenance.
format Online
Article
Text
id pubmed-8569008
institution National Center for Biotechnology Information
language English
publishDate 2021
publisher Nature Publishing Group UK
record_format MEDLINE/PubMed
spelling pubmed-85690082021-11-05 Design considerations for workflow management systems use in production genomics research and the clinic Ahmed, Azza E. Allen, Joshua M. Bhat, Tajesvi Burra, Prakruthi Fliege, Christina E. Hart, Steven N. Heldenbrand, Jacob R. Hudson, Matthew E. Istanto, Dave Deandre Kalmbach, Michael T. Kapraun, Gregory D. Kendig, Katherine I. Kendzior, Matthew Charles Klee, Eric W. Mattson, Nate Ross, Christian A. Sharif, Sami M. Venkatakrishnan, Ramshankar Fadlelmola, Faisal M. Mainzer, Liudmila S. Sci Rep Article The changing landscape of genomics research and clinical practice has created a need for computational pipelines capable of efficiently orchestrating complex analysis stages while handling large volumes of data across heterogeneous computational environments. Workflow Management Systems (WfMSs) are the software components employed to fill this gap. This work provides an approach and systematic evaluation of key features of popular bioinformatics WfMSs in use today: Nextflow, CWL, and WDL and some of their executors, along with Swift/T, a workflow manager commonly used in high-scale physics applications. We employed two use cases: a variant-calling genomic pipeline and a scalability-testing framework, where both were run locally, on an HPC cluster, and in the cloud. This allowed for evaluation of those four WfMSs in terms of language expressiveness, modularity, scalability, robustness, reproducibility, interoperability, ease of development, along with adoption and usage in research labs and healthcare settings. This article is trying to answer, which WfMS should be chosen for a given bioinformatics application regardless of analysis type?. The choice of a given WfMS is a function of both its intrinsic language and engine features. Within bioinformatics, where analysts are a mix of dry and wet lab scientists, the choice is also governed by collaborations and adoption within large consortia and technical support provided by the WfMS team/community. As the community and its needs continue to evolve along with computational infrastructure, WfMSs will also evolve, especially those with permissive licenses that allow commercial use. In much the same way as the dataflow paradigm and containerization are now well understood to be very useful in bioinformatics applications, we will continue to see innovations of tools and utilities for other purposes, like big data technologies, interoperability, and provenance. Nature Publishing Group UK 2021-11-04 /pmc/articles/PMC8569008/ /pubmed/34737383 http://dx.doi.org/10.1038/s41598-021-99288-8 Text en © The Author(s) 2021 https://creativecommons.org/licenses/by/4.0/Open AccessThis article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ (https://creativecommons.org/licenses/by/4.0/) .
spellingShingle Article
Ahmed, Azza E.
Allen, Joshua M.
Bhat, Tajesvi
Burra, Prakruthi
Fliege, Christina E.
Hart, Steven N.
Heldenbrand, Jacob R.
Hudson, Matthew E.
Istanto, Dave Deandre
Kalmbach, Michael T.
Kapraun, Gregory D.
Kendig, Katherine I.
Kendzior, Matthew Charles
Klee, Eric W.
Mattson, Nate
Ross, Christian A.
Sharif, Sami M.
Venkatakrishnan, Ramshankar
Fadlelmola, Faisal M.
Mainzer, Liudmila S.
Design considerations for workflow management systems use in production genomics research and the clinic
title Design considerations for workflow management systems use in production genomics research and the clinic
title_full Design considerations for workflow management systems use in production genomics research and the clinic
title_fullStr Design considerations for workflow management systems use in production genomics research and the clinic
title_full_unstemmed Design considerations for workflow management systems use in production genomics research and the clinic
title_short Design considerations for workflow management systems use in production genomics research and the clinic
title_sort design considerations for workflow management systems use in production genomics research and the clinic
topic Article
url https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8569008/
https://www.ncbi.nlm.nih.gov/pubmed/34737383
http://dx.doi.org/10.1038/s41598-021-99288-8
work_keys_str_mv AT ahmedazzae designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT allenjoshuam designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT bhattajesvi designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT burraprakruthi designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT fliegechristinae designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT hartstevenn designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT heldenbrandjacobr designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT hudsonmatthewe designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT istantodavedeandre designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT kalmbachmichaelt designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT kapraungregoryd designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT kendigkatherinei designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT kendziormatthewcharles designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT kleeericw designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT mattsonnate designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT rosschristiana designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT sharifsamim designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT venkatakrishnanramshankar designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT fadlelmolafaisalm designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic
AT mainzerliudmilas designconsiderationsforworkflowmanagementsystemsuseinproductiongenomicsresearchandtheclinic