Searches for the Standard Model Higgs Boson Decaying into $\mu^{+} \mu^{-}$ and New Phenomena at TeV Mass Scale with the ATLAS Detector

The greatest triumph of the Standard Model (SM) of particle physics over the past half century is the discovery of the Higgs boson by the ATLAS and CMS experiments at the Large Hadron Collider (LHC) in 2012 during the Run 1 program. The Higgs boson, predicted in the early 1960s, is the cornerstone of the SM. All the massive elementary particles acquire masses through the interactions with it. Different from the gauge field couplings, the Higgs couplings to different particles are dependent on their masses. After the discovery of the Higgs boson, one of the highest priorities of the LHC experiments is to measure its properties, particularly its non-uniform couplings. The Higgs signal strength together with its couplings measured in the bosonic decay channels ($WW$ and $ZZ$) have already achieved precision better than 15%. Observation of the Higgs decaying into fermion pairs is much harder due to larger background or smaller branching ratio (Br). During the Run 2 program (started in 2015) at the LHC operated with center-of-mass energy $\sqrt s$ =13 TeV, the Higgs decaying into the third generation of fermion pairs, $\tau \bar\tau$ and $b\bar b $, has been observed. But the Higgs boson decaying into the second generation of lighter fermions has not shown any evidence in the experiments at the LHC. This dissertation will present a search for the SM Higgs boson decaying into the dimuon final state. $H \rightarrow \mu\mu$ is a rare decay process but also a unique and relatively clean channel to study the Yukawa coupling between the Higgs boson and the second generation of fermions. However, the Br of $H \rightarrow \mu\mu$ is only $\sim$0.02%, and the irreducible dimuon event rate from the Drall-Yan process is three orders of magnitude higher. To improve the detection sensitivity, the multivariate analysis (MVA) method is incorporated for the first time in the $H \rightarrow \mu\mu$ search program. The output of the MVA, the Boosted Decision Trees (BDT) score, is used as the discriminant for event categorization to tag the $H \rightarrow \mu\mu$ event produced in the Vector-Boson Fusion (VBF) process. Using data collected by the ATLAS experiment in 2015 and 2016, corresponding to an integrated luminosity of 36.1 fb$^{-1}$, a much more sensitive upper limit on the Higgs boson production cross section times Br($H \rightarrow \mu\mu$) is set to be 3.0 times the SM prediction at 95% confidence level (CL) for the Higgs mass of 125 GeV. This result has been improved by a factor of more than two comparing with the previous Run 1 result. Despite its great success in providing experimental predictions for the particles and describing their interactions, the SM is not considered as the ultimate theory. Since many mysteries observed in the universe, such as dark matter and dark energy, cannot be explained by the SM. Therefore, searching for new physics beyond the SM (BSM) becomes a strong scientific driver for the LHC experiments. The second part of this dissertation will present studies of searching for new phenomena in high-mass scale with the dilepton final states, $\mu^+\mu^-$ and $e^+e^-$. The dilepton final states provide clean and robust signatures with relatively better-understood background to explore the new phenomena in the high-mass regime. The invariant mass spectra of the dilepton final states are used as the discriminants to search for resonance or broad excess as new physics signature. No deviation from the SM prediction is observed with 36.1 fb$^{-1}$ data. Much more stringent limits have been set on the parameters of interest for various models comparing with the Run 1 results. For new gauge bosons, $Z'_{SSM}$, $Z'_{\chi}$ and $Z'_{\psi}$ predicted in extended gauge theory models, the observed lower limits on mass have been set to be 4.5 TeV, 4.1 TeV and 3.8 TeV at 95% CL, respectively. While for the Contact Interactions, the observed lower limits on the corresponding energy scale $\Lambda$ are set between 24 TeV and 40 TeV depending on the different scenarios.