Search for Higgs boson pair production in the 𝑏𝑏𝜏𝜏 decay channel with the ATLAS experiment

In the Standard Model, the self-coupling of the Higgs boson is introduced by the electroweak symmetry breaking mechanism, the strength of which is related to the mass of Higgs boson and the coupling constant of weak interaction. At the Large Hadron Collider (LHC), Higgs boson pairs ($HH$) are dominantly produced through the self-coupling of Higgs bosons and the Yukawa coupling with top-quarks. The interference of these two processes suppresses the production cross section of the $HH$ to about one thousandth of that of the single Higgs boson. Nevertheless, many new physics models predict new scalar particles or anomalous couplings that can change the form of the Higgs potential and the strength of the self-coupling, thereby increasing the yield of $HH$ events, and making it possible to observe Higgs boson pair production with the data collected up to the present. The search for Higgs boson pair production is one of the key research topics of the ATLAS experiment in recent years since the discovery of the Higgs boson in 2012. It can improve our knowledge of the nature of the Higgs boson self-coupling and the Higgs potential, as well as deepen our understanding of the electroweak theory of the Standard Model and the thermal history of the early universe. In this thesis, a search for the Higgs boson pair production is performed in events with two $b$-jets and two $\tau$-leptons, by exploring the proton-proton collision data collected by the ATLAS detector at the LHC with a centre-of-mass energy of 13 TeV and an integrated luminosity of $139^{-1}$ fb. The $\tau_{\text{had}}\tau_{\text{had}}$ decay channel in which all $\tau$-leptons decay to hadrons is studied in detail, and the results are statistically combined with a semi-leptonic decay channel called the $\tau_{\text{lep}}\tau_{\text{had}}$ channel. Depending on the topology of the final-state physics objects, the analysis is divided into two types - the resolved and the boosted (a.k.a merged) analyses. In the resolved analysis, the two $b$-jets and two $\tau$-leptons can be reconstructed individually. Both the non-resonant and resonant $HH$ production modes are considered in this analysis. The signatures of the non-resonant mode are based on those of the Standard Model gluon-gluon fusion (ggF) and vector boson fusion (VBF) modes, while in the resonant production mode, the Higgs boson pairs are produced by the decay of heavy, narrow width, scalar particles, with mass below 1.6 TeV. In the search for non-resonant production mode, no significant excesses of events are observed above the expectations of the Standard Model background. Observed (expected) upper limits are placed at 95% confidence-level on the non-resonant $HH$ production cross section, which are 5.0 (4.4) times the expected ggF+VBF $HH$ production cross section considering only the $\tau_{\text{had}}\tau_{\text{had}}$ channel, and the corresponding results are 4.7 (3.9) after combining the $\tau_{\text{lep}}\tau_{\text{had}}$ channel. In addition, observed (expected) constraints are set on the strength of Higgs boson self-coupling in the analysis of non-resonant production mode. The values of self-coupling modifier $\kappa_{\lambda}$ that lie outside $[-2.5, 9.7]$ ($[-2.3, 9.8]$) are excluded at 95% confidence-level. The most significant deviation is observed in the search for resonant production mode at a resonance particle mass of 1 TeV, corresponding to a local (global) significance of $2.8\sigma$ ($1.7\sigma$) considering only the $\tau_{\text{had}}\tau_{\text{had}}$ channel, and $3.0\sigma$ ($2.0\sigma$) after combining the $\tau_{\text{lep}}\tau_{\text{had}}$ channel. At 95% confidence-level, the observed (expected) upper limits on the cross section of resonant $HH$ production are 27-1600 fb (18-1000 fb) considering only the $\tau_{\text{had}}\tau_{\text{had}}$ channel, and 23-930 fb (12-840 fb) after the combination. The actual value depends on the mass hypothesis of the resonance particle. In the boosted analysis, the two small radius jets merge into a single large radius jet, and the hadronic decay products of two $\tau$'s merge into a single object called Di-$\tau$, thus only the merged objects are reconstructed. This analysis focuses on the resonant $HH$ production, and it extends the search range of resonance mass to 3 TeV by using the Di-$\tau$ tagging technique. Finally, no significant excesses of events are observed with respect to the Standard Model background prediction. The observed (expected) upper limits on the cross section of resonant $HH$ production are placed, which are 28-94 fb (32-74 fb), depending on the mass hypothesis of the resonance particle. Besides, a novel method is proposed to classify the hadronic decay modes of $\tau$-leptons. The classification method based on a DeepSet neural network architecture improves both the classification efficiency and purity by nine percent compared with current method. It is promising to apply this new method in future physics analyses.