Unravelling the top-Higgs coupling with the ATLAS experiment at LHC

The discovery of the Higgs boson in 2012 was a major success of the Large Hadron Collider (LHC) at CERN. With larger data-sets collected at the LHC since then, precise measurements of the Higgs boson properties are possible. The Higgs boson production in association with a pair of top quarks, where Higgs decays into a pair of b-quarks, tt̄H(H→bb̄), allows direct measurement of the top-Yukawa coupling. The tt̄H(H→bb̄) process has a challenging final state with at least four b-jets, which requires a b-jet identification method known as b-tagging. In view of the operation of the ATLAS detector under High-Luminosity LHC conditions, the central tracking system will be upgraded to maintain high levels of performance. This thesis presents the performance of the b-tagging algorithms focusing on impact parameter-based and secondary vertex-based taggers using the updated ATLAS Inner Tracker (ITk) simulation. Some re-optimization of the b-taggers is performed, and the impact on performance is also presented. The tt̄H(H→bb̄) analysis is performed using 139 fb-1 of ATLAS Run 2 data at $\sqrt{s}$ = 13 TeV and takes advantage of the most recent object performance algorithms. The high b-jet multiplicity due to additional top quark decay products requires dedicated analysis strategies based on machine learning. The multivariate analysis approaches using Deep Neural Networks (DNNs) were developed to improve the search sensitivity while constraining large tt̄ + bb̄ background sub-components. This thesis compares DNN performance to that of previously used Boosted Decision Trees (BDTs). The signal strength is measured inclusively and differentially with respect to the Higgs boson transverse momentum using the Simplified Template Cross-Section formalism. The expected significance of the tt̄H signal over the expected SM background using DNN is 2.71 σ , compared to 2.54 σ using BDTs.