Searches for Higgs boson pair production in bbbb and bb𝛄𝛄 final states at Compact Muon Solenoid detector

The discovery of the Higgs boson makes the Standard Model (SM) a promising theory to understand fundamental physics. Despite its progress, the theory does not explain many observed phenomena such as dark matter and hierarchy problems. This motivates us to go beyond the Standard Model (BSM). Furthermore, the Higgs self-coupling is rather weakly constrained by the current measurements and allows the possibility of new physics. As the latest discovered piece of the SM, the Higgs boson can be used to explore new physics models. There are novel BSM resonances that directly couple to Higgs boson and are easier to observe with direct searches. The thesis focuses on non-resonant and resonant di-Higgs searches at the Large Hadron Collider (LHC), where a pair of Higgs bosons get produced during proton-proton collisions. In the first part of the work, we present di-Higgs searches at the high luminosity LHC (HL-LHC) in the final state of four bottom quarks at √s = 14 TeV. The study is performed using simulations of phase-2 Compact Muon Solenoid (CMS) detector assuming up to 200 proton-proton collisions within each bunch-crossing and 3 ab-1 total integrated luminosity. We start with the resonant di-Higgs production via vector boson fusion (VBF) in a boosted regime where resonance is a massive spin-2 bulk KK graviton particle predicted by the warped extra dimension model. This is the first CMS analysis that explores VBF resonant production mechanism. Both the Higgs bosons are required to be sufficiently Lorentz-boosted to reconstruct them as a large-area jet. The signal also contains two energetic VBF jets in the forward pseudorapidity regions of the detector. The unique topology of the production and decay would benefit from the upgraded phase-2 CMS detector having extended tracker coverage and a high granularity calorimeter which motivates this study. The SM multijet processes are the main background for the analysis. Expected signal significances for observing a bulk KK graviton, having a mass between 1.5–3.0 TeV and a width that is narrow up to 5%, are projected, assuming the signal cross-section to be 1 fb. Following a similar boosted strategy, non-resonant di-Higgs production for the SM and effective field theory (EFT) motivated shape benchmarks are also studied. A 95% confidence level (CL) upper limit on the product of Higgs boson pair production cross-section and branching fraction is presented for the benchmarks. The results show prospects of significant sensitivity for EFT motivated non-resonant di-Higgs production at the HL-LHC. In the second part, we use 2016, 2017, and 2018 LHC run period data of the CMS detector at √s = 13 TeV with 138 fb-1 total integrated luminosity and present the study for resonant di-Higgs production via the gluon-gluon fusion in the final state of two photons and two bottom quarks in a resolved regime. The physics is motivated by the warped extra dimension model where spin-0/2 resonance decays into two Higgs bosons and the next-to-minimal supersymmetric model where spin-0 resonance decays into a Higgs boson and another spin-0 particle different from the discovered Higgs boson. It is the first analysis that explores an NMSSM motivated scenario in this final state. The diphoton decay mode of the Higgs boson benefits from the excellent energy resolution of the CMS electromagnetic calorimeter, which makes this channel the most sensitive one among the various di-Higgs decay modes. The analysis uses machine learning methods to reject dominating diphoton QCD background contamination. As a result, it enhances the analysis sensitivity despite having a low di-Higgs branching fraction channel. With the narrow-width approximation, a model-independent 95% CL upper limit on the product of resonant production cross-section and branching fraction is set for resonance mass up to 1 TeV. The results are also compared with appropriate BSM predictions to exclude allowed resonance mass ranges.