Pair production of Higgs bosons in the final state with bottom quarks and $\tau$ leptons in the ATLAS experiment: Search results using LHC Run 2 data and prospect studies at the HL-LHC

After the discovery of the Higgs boson, the ultimate test of the electroweak symmetry breaking and the Standard Model (SM) of particle physics is to establish evidence of Higgs boson self-coupling, which can be achieved by searching for pair production of Higgs bosons. In addition, many theories beyond the SM predict heavy resonances that could decay into pairs of Higgs bosons. A search for non-resonant and resonant pair production of Higgs bosons in the final state with two bottom quarks and two $\tau$ leptons ($b\bar{b}\tau^+\tau^-$) is presented for 36.1 fb$^{-1}$ of proton-proton collision data at a centre-of-mass energy of 13 TeV recorded by the ATLAS experiment at the Large Hadron Collider (LHC). The observed (expected) 95% confidence level (CL) upper limit on the non-resonant Higgs boson pair production cross-section times the $b\bar{b}\tau^+\tau^-$ branching ratio corresponds to 12.7 (14.8) times the SM prediction. The ratio of the Higgs boson self-coupling to its SM expectation, $\kappa_{\lambda}$, is observed (expected) to be constrained to $\kappa_{\lambda}\in[-7.4, 15.7]$ $\left([-8.9, 16.8]\right)$ at 95% CL. The sensitivity is extrapolated to a search with 14 TeV centre-of-mass energy and 3000 fb$^{-1}$, which is the target integrated luminosity of the High-Luminosity LHC (HL-LHC). Various extrapolation assumptions are taken into account. The estimated expected signal significance for the SM Higgs boson pair production in the $b\bar{b}\tau^+\tau^-$ final state at the HL-LHC reaches 2.1 standard deviations, while the expected allowed $\kappa_{\lambda}$ interval is $\kappa_{\lambda}\in[-0.8, 8.8]$ at 95% CL, assuming the SM Higgs boson couplings. Furthermore, potential improvements of the analysis techniques are discussed in the context of searches for pair production of Higgs bosons with the full Run 2 ATLAS data.