Searches for Lepton Flavour Violating Decays of the Higgs Boson in pp Collisions with the ATLAS Detector

The Standard Model (SM) of particle physics accurately classifies elementary particles and describes their interactions based on experimental observations and theoretical predictions. However, it lacks explanations for certain phenomena like neutrino oscillations. Motivated by the search for a complete theory of nature, physicists aim to extend the SM by exploring experimental setups which may provide indications of new physics. One of the main goals of the Large Hadron Collider (LHC) experiments is to search for signals that violate the SM predictions, including the signal of the lepton-flavour-violating (LFV) decays of the Higgs boson. \par This thesis presents two direct searches for lepton flavour violation in Higgs boson decays, $H\rightarrow e\tau$ and $H\rightarrow\mu\tau$, using data from the ATLAS detector at the LHC. The analysis is based on the dataset of proton-proton collisions at a centre-of-mass energy $\sqrt{s}=13$ TeV, corresponding to an integrated luminosity of 138 fb$^{-1}$. \par Two background estimation techniques, the MC-template method and the symmetry method, are utilized in this analysis. The MC-template method relies on data-corrected simulation samples, while the symmetry method exploits the symmetry between electrons and muons in the SM backgrounds. \par This thesis focuses on the studies of the hadronic decays ($\tau\rightarrow$ hadrons $\nu_{\tau}$) of the $\tau$-lepton using the MC-template method. The symmetry method is also briefly described. Additionally, the leptonic decay ($\tau\rightarrow\ell\nu_{\ell}\nu_{\tau}$) channels and the combined results are described. \par No significant excess of events was observed in the searches for the lepton-flavour-violating decays of the Higgs boson. The results are interpreted as upper limits on the branching ratios, $\mathcal{B}(H\rightarrow e\tau)$ and $\mathcal{B}(H\rightarrow\mu\tau)$, determined at 95$\%$ confidence level. The observed (expected) branching ratio limits are $\mathcal{B}(H\rightarrow e\tau)<20\%$ (0.12$\%$) and $\mathcal{B}(H\rightarrow\mu\tau)<0.18\%$ (0.09$\%$), obtained using the MC-template method. The results obtained using the symmetry method are represented as the branching ratio difference, and the best-fit value of $\mathcal{B}(H\rightarrow\mu\tau)-\mathcal{B}(H\rightarrow e\tau)$, measured in the leptonic decay channels, is (0.25$\pm$0.10)$\%$, which is compatible with zero within a significance of 2.5$\sigma$. \par The prospect of the searches of LFV decays of the Higgs boson with 3000 fb$^{-1}$ data samples of proton-proton collisions at $\sqrt{s}=14$ TeV using the ATLAS detector at the High-luminosity LHC (HL-LHC) is presented. The expected HL-LHC results are estimated by extrapolating the dataset with integrated luminosity of 138 fb$^{-1}$. In comparison, the expected upper limits obtained using the HL-LHC data for the branching ratio $\mathcal{B}(H\rightarrow e\tau)$ and $\mathcal{B}(H\rightarrow\mu\tau)$ are more stringent by a factor of 4.8 and 3.9, respectively.