Experimental studies of charmonium production and $B_c$ mesons at LHCb

Studies of the properties of heavy-flavour hadrons can help deepen the understanding of strong interactions, which are described by quantum chromodynamics (QCD). QCD processes at short distances are perturbative, while those at long distances are non-perturbative and remain less understood. This dissertation presents the experimental studies of charmonia and $B_c$ mesons with data samples of proton-proton ($pp$) collisions collected by the LHCb detector at the Large Hadron Collider. In the production of quarkonium in $pp$ collisions, the creation of heavy-quark pairs is expected to be perturbative, while the subsequent evolution of the heavy-quark pairs to quarkonia is non-perturbative, for which the calculations rely essentially on experimental inputs. The inclusive production cross-sections of prompt $J/\psi$ mesons and those of $J/\psi$ mesons from $b$-hadron decays in $pp$ collisions at the centre-of-mass energy $\sqrt{s}=13{\mathrm{\ TeV}}$ are measured using a data sample corresponding to an integrated luminosity of $3.05{\mathrm{\ pb}^{-1}}$ collected in early 2015. The kinematic range of the measured $J/\psi$ mesons is ${p_{\mathrm{ T}}}<14{\mathrm{\ Ge\kern -0.1em V\!/}c}$ and $2.0 < y < 4.5$, where ${p_{\mathrm{ T}}}$ and $y$ indicate the transverse momentum and the rapidity of the $J/\psi$ meson. The double differential cross-sections as functions of ${p_{\mathrm{ T}}}$ and $y$ of the $J/\psi$ mesons are measured. The integrated cross-section of prompt $J/\psi$ mesons is determined to be $15.03\pm 0.03{\mathrm{\ (stat)}} \pm 0.91{\mathrm{\ (syst) \ {\rm \mu} b}}$, and that of $J/\psi$ mesons from $b$-hadron decays is $2.25\pm 0.01{\mathrm{\ (stat)}} \pm0.13{\mathrm{\ (syst)}} \ {\rm \mu} {\rm b}$. The results of prompt $J/\psi$ mesons are consistent with the calculations of non-relativistic QCD~(NRQCD) at the next-to-leading order~(NLO), and those of $J/\psi$ mesons from $b$-hadron decays are consistent with calculations of FONLL, the fixed order plus next-to-leading logarithms. The ratios of the production cross-sections at $\sqrt{s}=13{\mathrm{\ TeV}}$ and those at $\sqrt{s}=8{\mathrm{\ TeV}}$ are determined as functions of ${p_{\mathrm{ T}}}$ and $y$ to provide more precise comparisons with the theoretical models, since a large fraction of uncertainties cancel in the ratios from both the experimental and theoretical sides. The FONLL predictions tend to underestimate the cross-section ratios for $J/\psi$ mesons from $b$-hadron decays. Measurements of $J/\psi$ pair production can provide additional information to test the QCD models. The $J/\psi$ pair can be produced either through the process of single parton scattering (SPS), or through that of double parton scatterings (DPS). The SPS contribution can be calculated by QCD models, and the DPS contribution can be determined from measurements of prompt $J/\psi$ production cross-section. The production cross-sections of $J/\psi$ pairs in $pp$ collisions at $\sqrt{s}=13{\mathrm{\ TeV}}$ are measured as functions of various kinematic variables with a data sample corresponding to an integrated luminosity of $279{\mathrm{\ pb}^{-1}}$. The integrated production cross-section of $J/\psi$ pair for $J/\psi$ mesons in the kinematic range ${p_{\mathrm{ T}}}<10{\mathrm{\ GeV\!/}c}$ and $2.0 < y < 4.5$ is measured to be $15.2 \pm 1.0{\mathrm{\ (stat)}} \pm 0.9{\mathrm{\ (syst)}} {\mathrm{\ nb}}$. The contributions of SPS and DPS are studied under various theoretical assumptions. Significant DPS contributions are observed in the high $\Delta y$ range, where $\Delta y$ is the difference of the rapidities between the two $J/\psi$ mesons. The SPS contributions are consistent with the calculations of the leading-order (LO) $k_{\rm T}$ factorization, and are overestimated by the NLO colour-singlet model. The decays and spectroscopy of $B_c$ mesons are described by various QCD models, which can be tested through experimental studies of $B_c$ mesons. The branching fraction ratio $\mathcal{B}(B_c^{+} \to \psi(2S) \pi^{+})/\mathcal{B}(B_c^{+} \to J/\psi \pi^{+})$ is measured to be $0.268\pm0.032{\mathrm{\ (stat)}}\pm0.007{\mathrm{\ (syst)}}\pm0.006(\mathcal{B})$ with the data samples collected in 2011 and 2012, corresponding to integrated luminosities of $1 {\mathrm{\ fb}^{-1}}$ at $\sqrt{s}=7{\mathrm{\ TeV}}$ and $2 {\mathrm{\ fb}^{-1}}$ at $\sqrt{s}=8{\mathrm{\ TeV}}$, respectively. The last uncertainty is due to the uncertainties of the branching fractions of the $J/\psi \to \mu^{+}\mu^{-}$ and $\psi(2S) \to \mu^{+}\mu^{-}$ decays. The result agrees well with the calculations of the NRQCD approach at NLO and the $k_{\rm T}$ factorization method. The $B_c(2S)^+$ state, which was observed by the ATLAS experiment, is searched for in the $B_c^{+}\pi^{+}\pi^{-}$ mass spectrum using the data sample collected in 2012. No signal is observed. Upper limits are set on the product of the relative production cross-section $\sigma(B_c(2S)^+)/\sigma(B_c^{+})$ and the branching fraction $\mathcal{B}(B_c(2S)^+\to B_c^{+}\pi^{+}\pi^{-})$ under different assumptions of the $B_c(2S)^+$ mass. The upper limits agree with the ATLAS result only if the unpublished efficiency of reconstructing the $B_c(2S)^+$ state relative to that for the $B_c^{+}$ meson is very large at ATLAS.