Measurement of the Higgs trilinear self-coupling from the $\mathrm{HH}\rightarrow \mathrm{b\bar{b}}\gamma\gamma$ and $\mathrm{t\bar{t}H}(\gamma\gamma)$ channels with the CMS detector

This thesis presents a search for the non-resonant production of Higgs boson pairs in the final state with one Higgs boson decaying into a pair of $\mathrm{b\bar{b}}$ quarks and the other into a pair of photons ($\mathrm{HH}\rightarrow \mathrm{b\bar{b}}\gamma\gamma$). The data were collected by the CMS detector in proton-proton collisions with a center-of-mass energy of 13~TeV, for a total integrated luminosity of 137~fb$^{-1}$. The measurement of the HH production experimentally constrains the Higgs boson trilinear self-coupling $\lambda_{HHH}$, and the Higgs boson coupling to the top quark, $y_\mathrm{t}$, and probes the presence of phenomena beyond the Standard Model. The constraints on the $\lambda_{HHH}$ and on the $y_t$ parameters are significantly improved through the simultaneous fit of the cross sections of the HH processes and of the associated production of the Higgs boson with a $\mathrm{t\bar{t}}$ quark pair, with the Higgs boson decaying in two photons $\mathrm{t\bar{t}H}(\gamma\gamma)$. Such processes are intrinsically correlated because they both depend from the $\lambda_{HHH}$ and $y_t$ constants. The $\mathrm{HH}\rightarrow \mathrm{b\bar{b}}\gamma\gamma$ experimental signature, similar to $\mathrm{t\bar{t}H}(\gamma\gamma)$, consists in two heavy-flavor high-energy jets (b-jets) and two high-energy photons. The HH signal can be identified as a peak in the diphoton invariant mass distribution, and a peak in the invariant mass of the two b-jets, at the value of the Higgs boson mass. The resolution on the photon energy is a key element to achieve the best sensitivity because it determines the width of the $\mathrm{H}\rightarrow \gamma\gamma$ peak in the diphoton mass distribution. For this reason, a work is carried out to optimize the performance of the CMS electromagnetic calorimeter (ECAL), which provides the energy measurements for photons (and electrons). In particular, the electrons from the W and Z bosons decays are used to calibrate the relative response of the ECAL channels and to correct for the response variations in time. At the analysis level, exclusive categories targeting the specific $\mathrm{t\bar{t}H}$ or HH events are defined to maximize the sensitivity to the HH and $\mathrm{t\bar{t}H}$ signals. Multivariate techniques are used to optimize the separation of each signal from the background events, as well as the separation between the HH and the $\mathrm{t\bar{t}H}$ signals. No significant deviations from the SM predictions are found for the $\mathrm{HH}\rightarrow \mathrm{b\bar{b}}\gamma\gamma$ process. In particular, no evidence of the HH signal is found. The observed (expected) upper limit on the inclusive $\mathrm{HH}\rightarrow \mathrm{b\bar{b}}\gamma\gamma$ cross section is 7.7 (5.2) times the Standard Model. Constraints on anomalous values of the Higgs coupling parameters $\kappa_\lambda=\lambda_{HHH}/\lambda_{HHH}^{SM}$ and $\kappa_t=y_t/y_t^{SM}$ are also extracted. In the hypothesis of no HH signal and all the other Higgs couplings equal to their SM value, the $\kappa_\lambda$ parameter is constrained in the range $[-3.3, 8.5]$ at $95\% $ confidence level. A two dimensional likelihood scan of the $\kappa_\lambda$ and $\kappa_t$ parameters exploiting the constraints from the HH and the single Higgs boson, especially $\mathrm{t\bar{t}H}$, processes is performed. The best-fit values of the ($\kappa_\lambda$, $\kappa_t$) parameters are consistent with the SM within the uncertainties. In addition, the prospects for the HH search at the high-luminosity LHC (HL-LHC) are studied for a center-of-mass energy of 14~TeV and an integrated luminosity of 3000~fb$^{-1}$ using a parametric simulation of the upgraded CMS detector under different performance scenarios. The analysis shows that a 3$\sigma$ observation of the HH process is in the reach of the HL-LHC.