Experimental Study of Exotic Hadrons at LHCb

Among more than twenty tetraquark candidates that contain at least one $c\bar{c}$ pair, the electrically charged states, $e.g.$ $Z_c(3900)^+$ and $Z_c(4430)^+$, are smoking-gun evidence of tetraquark states, since their minimum-constituent-quark contents are $c\bar{c}u\bar{d}$. Apart from the $c\bar{c}$ pair, at least a pair of quarks with different flavors, $u\bar{d}$, exists. This dissertation used the full LHCb data sample corresponding to an integrated luminosity of $9{fb}^{-1}$ to search for the strangeness partner $(c\bar{c}u\bar{s})$, $i.e.$ $Z_{cs}^+$. A six-dimensional amplitude analysis of the $B^{+}\to J/\psi \phi K^+$ decay is performed, and two charged tetraquark states that contain an $s$ quark in their quark contents, $Z_{cs}(4000)^+$和$Z_{cs}(4220)^+$, are observed in the $j/\psi K^+$ final state. The mass and width of the former are $4003\pm6_{-14}^{+\,\,\,4} MeV$ and $131\pm {15}\pm26 MeV$, respectively, and the spin-parity quantum numbers are $J^P=1^+$; the mass and width of the latter are $4219\pm24_{-30}^{+43} MeV$ and $233\pm {52}_{-73}^{+97} MeV$, respectively, and the spin and parity is not conclusive. The first uncertainties are statistical and the second systematic. In addition, apart from the four exotic states observed in the $J/\psi\phi$ combination in 2016, two new states, $X(4685)$ and $X(4630)$, are observed in this analysis, and their quantum numbers are determined to be $J^{PC}=1^{++}$. The same data sample was also used to study the $\Lambda^0_{b}\to J/\psi p K^{-}$ decay, in order to better determine the properties of the pentaquark states observed by the LHCb experiment in 2015. The event selection is significantly improved with respect to the 2015 analysis. The selection efficiency improves by a factor of two under the same purity, and the number of $\Lambda^0_{b}$ signals increases by a factor of nine. This great improvement leads to the observation of a new narrow pentaquark $P_{c}(4312)^+$ and the fine structure of the previously observed $P_{c}(4450)^+$, where two narrow peaks exist, $P_{c}(4440)^+$ and $P_{c}(4457)^+$. The masses and widths of these three new states are \begin{equation*} \begin{split} &m=4311.9\pm{0.7}_{-0.6}^{+6.8} MeV \qquad \Gamma=9.8\pm {2.7}_{-4.5}^{+3.7} MeV ,\\ &m=4440.3\pm{1.3}_{-4.7}^{+4.1} MeV \qquad \Gamma=20.6\pm {4.9}_{-10.1}^{\,+\,\,8.7} MeV ,\\ &m=4457.3\pm{0.6}_{-1.7}^{+4.1} MeV \qquad \Gamma=6.4\pm {2.0}_{-1.9}^{+5.7} MeV, \end{split} \end{equation*} where the first uncertainties are statistical and the second systematic. The masses of $P_{c}(4312)^+$ and $P_{c}(4457)^+$ locate around the $\Sigma_{c}\bar{D}$ and $\Sigma_{c}\bar{D}^{*}$ thresholds respectively, and they are consistent with the hypotheses of molecular models. In addition, a preliminary study of the Cabibbo-suppressed decay $\Lambda^0_{b} \to J/\psi p \pi^{-}$ was also performed. An evidence of pentaquark states is observed. A new decay $\Lambda^0_{b} \to \Lambda^+_{c} K^{+} K^{-} \pi^{-}$ is observed using a data sample corresponding to an integrated luminosity of $3{fb}^{-1}$. The branching fraction ratio with respect to the $\Lambda^0_{b} \to \Lambda^+_{c} D_{s}^{-}$ decay is measured to be $B( \Lambda^0_{b} \to \Lambda^+_{c} K^{+} K^{-} \pi^{-} )/B( \Lambda^0_{b} \to \Lambda^+_{c} D_{s}^{-})=(9.26 \pm 0.29 \pm 0.46 \pm 0.26)\times10^{-2}$, where the first uncertainty is statistical, the second is systematic, and the third is due to the uncertainty of the branching fraction $B( D_{s}^{-} \to K^{+} K^{-} \pi^{-} )$. With a larger data sample, this decay can be used to search for open-charm pentaquark states in the $\Lambda^+_{c} K^{+}$ final state.