Search for non-resonant Higgs pair production in the $b\bar{b}b\bar{b}$ final state with the ATLAS detector

The Standard Model (SM) of particle physics succeeds to describe the origin of particle mass by introducing the electroweak symmetry breaking (EWSB). However, we have not yet experimentally proven the actual structure of the Higgs potential, which is responsible for the EWSB. Several types of the Higgs potential are predicted in new physics scenarios such as electroweak baryogenesis. The Higgs potential can be determined only by measuring the Higgs self-couplings. I thus focus on the trilinear Higgs self-couplings, which is a coupling constant of the interaction of three Higgs bosons. This thesis presents a search for non-resonant Higgs pair production in the $b\bar{b}b\bar{b}$ final states $\left(pp{\rightarrow}HH{\rightarrow}b\bar{b}b\bar{b}\right)$ using 129 $\rm{fb}^{-1}$ of proton–proton collisions at a centre-of-mass energy of $\sqrt{s}=13$ TeV taken with the ATLAS detector. $HH$ production via the two leading production modes, gluon-gluon fusion (ggF) and vector-boson fusion (VBF), are sensitive to the trilinear Higgs self-coupling $\left(\kappa_\lambda\right)$ and the two Higgs bosons and two vector bosons $\left(HHVV\right)$ coupling $\left(\kappa_{2V}\right)$, where $\kappa_\lambda$ and $\kappa_{2V}$ are defined as the coupling ratios with respect to the SM predictions. The $b\bar{b}b\bar{b}$ final state, where both Higgs bosons decay to a pair of $b-$ and $\bar{b}-$quarks, is one of the most sensitive channels thanks to the highest branching ratio. On the other hand, the search in the $b\bar{b}b\bar{b}$ final state is challenging due to a huge amount of QCD multijet background. I developed a new analysis for the non-resonant $HH{\rightarrow}b\bar{b}b\bar{b}$. Two orthogonal selections targeting the ggF and VBF production are provided to increase the sensitivity for each production mode. I additionally adopted analysis categorizations in both selections to improve the sensitivity. The crucial key in this analysis is background estimation, because it is hard to model QCD multijet background in simulation. I thus utilized a fully data-driven approach using a novel neural network to properly estimate the background. No evidence for the $HH$ production is found, and the exclusion limits at 95% confidence level are set on the signal strength, which is defined as the ratio of the observed cross-section to the SM prediction, of the ggF$+$VBF $HH$ production cross-section, the trilinear Higgs self-coupling and the $HHVV$ coupling. The observed (expected) upper limit on the signal strength is set to 5.4 (8.1). The observed (expected) allowed region on the trilinear Higgs self-coupling is $\kappa_\lambda$$\in[-3.9, 11.1]$ $\left([-4.6, 10.8]\right)$. The observed (expected) allowed region on the $HHVV$ coupling is $\kappa_{2V}$$\in[-0.03, 2.11]$ $\left([-0.05, 2.12]\right)$. These results are consistent with the SM prediction $\left(\kappa_\lambda=\kappa_{2V}=1\right)$. They are the first results derived from the analysis targeting non-resonant $HH{\rightarrow}b\bar{b}b\bar{b}$ and improve these constraints by a factor of 2-4 with respect to the previous analyses. This thesis also presents the interpretation results using Effective Field Theory frameworks. This thesis additionally presents a statistical combination of three $HH$ analyses in the $b\bar{b}b\bar{b}$, $b\bar{b}\gamma\gamma$ and $b\bar{b}\tau\tau$ final states with up to 139 $\rm{fb}^{-1}$ of proton–proton collisions at $\sqrt{s}=13$ TeV in the ATLAS experiment. The $b\bar{b}b\bar{b}$, $b\bar{b}\gamma\gamma$ and $b\bar{b}\tau\tau$ final states provide the highest sensitivity for the signal strength, the trilinear Higgs self-coupling and the $HHVV$ coupling, and the statistical combination improves the constraints. The observed (expected) upper limit on the signal strength is 2.4 (2.9). The allowed region on the trilinear Higgs self-coupling is $\kappa_\lambda\in[-0.6, 6.6]$ $\left([-1.0, 7.1]\right)$ and on the $HHVV$ coupling is $\kappa_{2V}\in[0.07, 2.03]$ $\left([0.02, 2.06]\right)$.