Experimental study of hadron spectroscopy in beauty meson to open charm decays at LHCb

Quantum chromodynamics~(QCD) is the fundamental theory describing the strong interaction in the Standard Model~(SM) of particle physics. It has an important feature called the asymptotic freedom. The QCD coupling asymptotically goes to zero at high energy scale and thus the perturbative method is applicable, while the strong coupling in low energy region results in a failure of the perturbative method and makes the QCD calculation extremely difficult. Study of hadron spectroscopy is at the frontier of modern particle physics to promote the understanding of the nonperturbative property of the low energy QCD. Heavy-flavor hadrons, especially the exotic states containing heavy-flavor quark(s), provide an unique platform for the studies of the nonperturbative QCD. This thesis presents the first analyses of two $B$-meson decays to open-charm final state, $B^0\to D^+D^-K^+\pi^-$ and $B^+\to D_s^+D_s^-K^+$, using the proton--proton collision data collected by the Large Hadron Collider beauty~(LHCb) experiment at CERN. Two new mesons, the charm-strange meson $D_{s0}(2590)^+$ and the tetraquark candidate $X(3960)$, are observed in the $D^+K^+\pi^-$ and $D_s^+D_s^-$ systems, respectively. Amplitude analyses are performed to measure the properties of the two states. The $B^0\to D^+D^-K^+\pi^-$ decay is studied with the $pp$ collision data collected at a center-of-mass energy of $13\,\text{TeV}$ during the 2016--2018 period. The integrated luminosity of $5.4\,\text{fb}^{-1}$ Two new peaking structures in the $D^+K^+\pi^-$ mass spectrum are observed, corresponding to undiscovered excited. $D_s^+$ mesons. Due to the complexity of the four-body $B$-meson decay, this thesis only focuses on the study of the state near the $D^+K^+\pi^-$ mass threshold, which can be isolated from the messy structures at the higher mass by requiring the $K^+\pi^-$ system in the low mass region, $m(K^+\pi^-)<0.75\,\text{GeV}$. The statistical significance of the new $D_s^+$ state is found over 10 standard deviations~($\sigma$). An amplitude analysis is performed on the $B^0\to D^+D^-K^+\pi^-$ data in the low $K^+\pi^-$ mass region to measure the properties of the new $D_s^+$ state, whose mass and width, and spin-parity are measured to be $m=2591 \pm 6 \pm 7\,\text{MeV}$, $\Gamma=89 \pm 16 \pm 12\,\text{MeV}$ and $J^P=0^-$. The new resonance, denoted as $D_{s0}(2590)^+$, is a strong candidate to be the $D_s(2^1S_0)^+$ state, the radial excitation of the pseudoscalar ground-state $D_s^+$ meson. A large discrepancy between the measured mass of the $D_{s0}(2590)^+$ resonance and theoretical mass of the $D_s(2^{1} S_{0})^+$ state, arousing the interests of theorists to study the inner structure of the $D_{s0}(2590)^+$ meson. The $B^+\to D_s^+D_s^-K^+$ decay is observed for the first time using the full LHCb dataset collected in 2011, 2012 and 2015--2018 at center-of-mass energies of 7, 8 and 13\,TeV, respectively. The corresponding integrated luminosity is roughly $9\,\text{fb}^{-1}$. The branching fraction of the $B^+\to D_s^+D_s^-K^+$ decay is measured relative to that of the $B^+\to D^+D^-K^+$ process as $B(B^+\to D_s^+D_s^-K^+)/B(B^+\to D^+D^-K^+) = 0.525 \pm 0.033 \pm 0.027\pm 0.034$. A near-threshold $X(3960)$ is observed in the $D_s^+D_s^-$ mass spectrum with statistical significance over $10\sigma$. The mass, width and spin-parity of the $X(3960)$ state are measured with an amplitude analysis to be $m=3955\pm6\pm11\,\text{MeV}$, $\Gamma=48\pm17\pm10\,\text{MeV}$ and $J^{PC}=0^{++}$} The mass and width of the $X(3960)$ state are comparable to those of $\chi_{c0}(3930)$ meson previously observed in the $D^+D^-$ final state, and they both have spin-parity $J^{PC}=0^{++}$, indicating that they are an identical state, denoted as $X$. Under this assumption, the partial width fraction ratio of the two decay modes is obtained, ${\Gamma(X\to D^+D^-) }/{\Gamma(X\to D_s^+D_s^-)} = 0.29 \pm 0.09\pm 0.10\pm 0.08$. The partial width of the $X\to D_s^+D_s^-$ process is larger than that of the $X\to D^+D^-$ decay, indicating the exotic nature of the $X(3960)/\chi_{c0}(3930)$ state. In all the numerical results presented above, the first uncertainty is statistical, the second systematic, and the third~(optional) is due to the uncertainties on the input parameters, $e.g.$ branching fractions of the $D^+$ and $D_s^+$ decays.