Higgs-boson-pair production $H(\rightarrow b\overline{b})H(\rightarrow\gamma\gamma)$ from gluon fusion at the HL-LHC and HL-100 TeV hadron collider

We perform the most up-to-date comprehensive signal-background analysis for Higgs-pair production in $HH \to b\bar b \gamma\gamma$ channel at the HL-LHC and HL-100 TeV hadron collider, with the goal of probing the self-coupling $\lambda_{3H}$ of the Higgs boson. We simulate all the standard-model signal and background processes with the simulation tools almost as sophisticated as what experimentalists are using. We find that even for the most promising channel $HH \to b\bar{b}\gamma\gamma$ at the HL-LHC with a luminosity of 3000 fb$^{-1}$, the significance is still not high enough to establish the Higgs self-coupling at the standard model (SM) value. Instead, we can only constrain the self-coupling to $-1.0 < \lambda_{3H} < 7.6 $ at $ 95\% $ confidence level after considering the uncertainties associated with the top-Yukawa coupling and the estimation of backgrounds. Here we also extend the study to the HL-100 TeV hadron collider. With a luminosity of 3 ab$^{-1}$, we find there exists a bulk region of $2.6 \lesssim \lambda_{3H} \lesssim 4.8 $ in which one cannot pin down the trilinear coupling. Otherwise one can measure the coupling with a high precision. At the SM value, for example, we show that the coupling can be measured with about 20 \% accuracy. While assuming 30 ab$^{-1}$, the bulk region reduces to $3.1 \lesssim \lambda_{3H} \lesssim 4.3$ and the trilinear coupling can be measured with about 7 \% accuracy at the SM value. With all the simulated background events and results, our study can be useful to probe the Higgs potential of various models with extended Higgs sector, such as two-Higgs-doublet model, MSSM, etc.