Search for the dimuon decay of the Higgs boson in $139~\mathrm{fb}^{-1}$ of $pp$ collisions at $\sqrt{s}=13~\mathrm{TeV}$ with the ATLAS detector

The Standard Model (SM) of particle physics explains various phenomena observed so far. However, there are phenomena of the Higgs boson which are predicted in the SM but not yet confirmed experimentally. One of the Higgs boson's phenomena not yet well verified is the Yukawa interaction. It is assumed that ``charged fermions in all generations obtain mass proportional to each coupling strength through Yukawa interaction with the Higgs field''. Couplings of the Higgs boson to third-generation charged fermions have been observed. On the other hand, the couplings to first- and second-generation fermions have not been observed yet. This thesis presents a search for a dimuon decay of the Higgs boson. This decay contains the Yukawa interaction of the Higgs boson and the muon. The muon is one of the second-generation fermions in the SM. The dimuon decay of the Higgs boson provides a unique probe of the coupling between the Higgs boson and the second-generation fermions thanks to a more clean signature of the muons compared to the charm and strange quarks. Search for this decay is very challenging due to the huge $Z\rightarrow\mu\mu$ background events, and this decay has not been observed in previous studies using $\sqrt{s}=13~\mathrm{TeV}$ $pp$ collisions data collected in 2015 and 2016 at the Large Hadron Collider (LHC). This analysis uses $\sqrt{s}=13~\mathrm{TeV}$ $pp$ collisions data collected with the ATLAS detector at LHC in 2015--2018. The integrated luminosity is $139~\mathrm{fb}^{-1}$. The dimuon invariant mass is reconstructed and a peak around 125 GeV (the Higgs boson mass) above the irreducible background is searched for. In order to increase the sensitivity of the search, the improvements in the invariant mass resolution and the separation of the signal events from the background events are crucial. To improve the mass resolution, the photon from the QED final state radiation (FSR) of a muon is taken into account. In the events with the FSR, the photon carries away the energy of the muon, resulting in a smaller dimuon invariant mass. By reconstructing FSR photons and including the four-momentum in the invariant mass calculation, the number of signal events in the invariant mass region 120--130 GeV was increased by $1.4\%$ and the mass resolution was improved by $2.8\%$. To separate signal from background, the events are divided into 20 categories, based on the different final state particles in each Higgs production process, using multivariate analysis techniques. The invariant mass distribution of the background events is modeled using the Leading Order Drell-Yan analytic line-shape multiplied by empirical functions to account for imperfect modeling by the line shape. The observed signal significance is 2.0 standard deviations with respect to the background-only hypothesis. The signal strength, defined as the ratio of the observed signal yield to the expected one in the SM, is $1.2\pm0.6$. The result is consistent with the SM within the uncertainty. The CMS experiment reported an observed signal significance of 3 standard deviations. It is compatible with the result described in this thesis. Both results indicate that the origin of the muon mass arises from the Higgs mechanism. The results constitute the first evidence of the coupling of the Higgs boson to second-generation fermions. At present, the measurement of the signal strength is dominated by the statistical uncertainty. I expect that the Yukawa coupling to the muon is verified using higher statistics at LHC in the near future.