Search for Higgs pair production at LHC collider (CERN): The first measurement for Higgs potential and search for new physics

From the discovery of the Higgs boson in 2012, most of its properties such as mass, spin, production cross-section and its coupling to fermions and bosons have been measured. However, the trilinear self-coupling $\lambda_{HHH}$ of the Higgs boson has not been measured yet. This parameter controls the shape of the Higgs potential, explaining the importance of its measurement. Deviation from its Standard Model (SM) predicted value would indicate new physics beyond the SM (BSM). Deviations are quantified through the $\kappa_{\lambda}$ modifier. At the LHC, it is measured through the rate of the rare Higgs boson pair production (HH) process, which is the only direct way to access it. This thesis presents the search for Higgs boson pair production in its $H(\to\gamma\gamma)H(\to\bar{b}b)$ decay channel. The search is performed with the full Run-2 proton-proton collision data collected with the ATLAS detector at the LHC between 2015 and 2018 at a centre-of-mass energy of $\sqrt{s} = $ 13 TeV. This amount of data corresponds to an integrated luminosity of 139 fb$^{-1}$. Despite low statistics, the $\gamma\gamma\bar{b}b$ channel profits from a very clean experimental signature, an excellent $m_{\gamma\gamma}$ mass resolution, a smooth background and a large H$\to\bar{b}b$ branching ratio. Therefore, It is an excellent channel to perform $\kappa_{\lambda}$ measurement. Photon objects are so crucial for this channel that work concerning the improvement of their identification efficiency is presented in this thesis. A new algorithm based on Convolutional Neural Network (CNN) is implemented to improve identification efficiency. The CNN algorithm uses images from the electromagnetic calorimeter layers to learn the shower development of prompt photons and separates them from photons originating from jet fragmentation (fake photons). In addition, the mis-modelling of the shower shape in the simulation is corrected by redistributing the energy between the cluster cells to become consistent with the real data. The sensitivity to the $HH\to\gamma\gamma\bar{b}b$ signal is improved using a specific $b$-jet calibration technique. The signal is separated from the single Higgs boson and the continuum $\gamma\gamma$+jets backgrounds through a Multivariate Analysis (MVA). Additional tools are developed to improve the analysis sensitivity. They are not included in the publication of ATLAS collaboration presented in this thesis, but they would be used for the next analysis. No significant excess of $HH\to\gamma\gamma\bar{b}b$ events is observed in the data with respect to the expected background. Thus, an upper limit on the HH cross-section is set. The observed (expected) 95% confidence level (CL) limit corresponds to 4.1 (5.5) times the cross-section predicted by the SM. The observed (expected) 95% CL limit on the $\kappa_{\lambda}$ modifier is also derived to be [-1.5, 6.7] ([-2.4, 7.7]). An extrapolation of this result to Run-3 and to High-Luminosity LHC (HL-LHC) is also presented in this thesis.