Search for nonresonant Higgs boson pair production in final states with two bottom quarks and two photons in pp collisions at $\sqrt{s}=13\,\mathrm{TeV}$

A search for nonresonant production of Higgs boson pairs via gluon-gluon fusion and vector boson fusion in final states with two bottom quarks and two photons is presented. This search uses data from proton-proton collisions at a center-of-mass energy of ${\sqrt{s}=13\,\mathrm{TeV}}$ recorded by the CMS detector at the LHC from 2016 to 2018, corresponding to an integrated luminosity of ${137\,\mathrm{fb}^{-1}}$. No signal is observed, and a 95$\%$ confidence level upper limit is set on the product of the inclusive Higgs boson pair production cross section and branching fraction into $\gamma\gamma\mathrm{ b\bar{b}}$. The observed (expected) upper limit is determined to be $0.67$ ($0.45$)$\ \mathrm{fb}$, which corresponds to $7.7$ ($5.2$) times the standard model prediction. Assuming all other Higgs boson couplings are equal to their values in the standard model, the coupling modifiers of the trilinear self-coupling $\kappa_{\lambda}$ and the coupling between a pair of Higgs bosons and a pair of vector bosons $c_{2V}$ are constrained within the ranges $-3.3$ $<$ $\kappa_{\lambda}$ $<$ $8.5$ and $-1.3$ $<$ $c_{2V}$ $<$ $3.5$ at 95$\%$ confidence level. Constraints on $\kappa_{\lambda}$ are also set by combining this analysis with a search for single Higgs bosons produced in association with top quark-antiquark pairs, and by performing a simultaneous fit of $\kappa_{\lambda}$ and the top Yukawa coupling modifier $\kappa_{\mathrm{t}}$.