Aspects of Rare Production and Decay Processes of the Higgs Boson

<!--HTML-->The Higgs boson is the most recent fundamental particle to be experimentally observed. Due to its large mass and its couplings to massive bosons and fermions, it has a wide variety of decay channels. Among them is the still unobserved rare decay $H$ → $Z$($\ell$$^{+}$$\ell$$^{−}$)$\gamma$. In this thesis, a search for the $Z$$\gamma$ decay mode of the Standard Model Higgs boson as well as potential new higher-mass resonances decaying to the same final state is presented. The analysis is based on a dataset of 36.1 fb$^{−1}$ of proton-proton collisions collected with the ATLAS detector at the CERN Large Hadron Collider at a centre-of-mass energy of 13 TeV. No significant excess of events over the background-only expectation was observed, and consequently, an upper limit at the 95% CL on the cross section times branching ratio of 6.6 times the Standard Model expectation was derived. The expected upper limits were determined to be 5.2 (4.4) times the Standard Model expectation when assuming the presence (absence) of this decay mode.<p><p>An alternative Higgs boson decay process that creates the same final state is $H$ → $\gamma$$^{*}$($\ell$$^{+}$$\ell$$^{-}$)$\gamma$. In this process, the leptons are predominantly produced with much smaller invariant masses than in the $H$ → $Z$($\ell$$^{+}$$\ell$$^{−}$)$\gamma$ decay. This results in reduced opening angles between the leptons, especially in the final state with electrons. In this thesis, a preliminary, blinded analysis targeting the $H$ → $\gamma$$^{*}$($\ell$$^{+}$$\ell$$^{-}$)$\gamma$ decay with the ATLAS detector is presented for the first time. To retain events with close-by electron pairs, a dedicated identification algorithm was developed for such configurations. For an integrated luminosity of 80 fb$^{−1}$ and an LHC centre-of-mass energy of 13 TeV, a significance of 0.9σ and upper limits at the 95% CL on the cross section times branching ratio of 2.9 (2.1) times the Standard Model prediction are expected when assuming the absence (presence) of this decay mode.<p><p>At a proton-proton collider, the Higgs boson can be produced via different production processes. About 1% of the Higgs production cross section at the LHC stems from the production in association with a pair of top quarks. This process is very interesting because it gives tree-level access to the top quark Yukawa coupling. The $t\bar{t}$ $H$($b\bar{b})$ final state is complicated to analyse because of the presence of multiple jets and $b$-tags. One of its major backgrounds is $t\bar{t}c\bar{c}$ production, which is theoretically challenging in its own right. A portion of this thesis consists of studies on the Monte Carlo simulation of $t\bar{t}c\bar{c}$ production with massive charm quarks included in the matrix elements.