Study of the rare $B_s^0\to\phi\mu^+\mu^-$ and $B_s^0\to f_2'(1525)\mu^+\mu^-$ decays

The Standard Model of particle physics (SM) describes the microcosm of nature with great success. However, observations of phenomena not included in the SM indicate that the SM is not complete and motivate searches for New Physics (NP). An excellent laboratory for NP searches are precision measurements of rare $B$ decays, which gained particular interest with the appearance of the so-called flavour anomalies---a series of deviations between measurements and SM predictions. One of these flavour anomalies is the branching fraction $\mathcal{B}(B_s^0\to\phi\mu^+\mu^-)$, measured with data taken during 2011--2012 with the LHCb detector, which was found to be around $3$ standard deviations ($\sigma$) lower compared to the SM expectation at small invariant dimuon mass squared, $q^2$. This thesis presents the measurement of $\mathcal{B}(B_s^0\to\phi\mu^+\mu^-)$ using the full LHCb data sample collected during 2011--2018. The branching fraction is measured relative to the high-yield $B_s^0\to J/\psi\phi$ mode and in intervals of $q^2$, and the data are selected using kinematic and particle identification variables, including a multi-variate classifier. The signal decays are extracted from data using an unbinned maximum likelihood fit, and the selection efficiency is evaluated on simulation of the signal decays, to which data-driven corrections are applied to ensure a good description of data. The resulting branching fraction is measured as $\mathcal{B}(B_s^0\to\phi\mu^+\mu^-) = (8.14 \pm 0.21 \pm 0.16\pm 0.39\pm 0.03)\times10^{-7}$ with (in order) the statistical uncertainty, the systematic uncertainty, the uncertainty from the branching fraction $\mathcal{B}(B_s^0\to J/\psi\phi )$, and an uncertainty from the extrapolation to the full $q^2$ range. In the $q^2$ range $1.1 < q^2 < 6.0\,\text{GeV}^2 \!/c^4$, the branching fraction is found to be $3.6\,\sigma$ below a prediction based on lattice QCD and light-cone sum rule (LCSR) calculations and $1.8\,\sigma$ lower compared to LCSR calculations only. This thesis further presents the first observation of the $B_s^0\to f'_2(1525)\mu^+\mu^-$ decay and a measurement of its branching fraction. The analysis is performed similarly to the measurement of $B_s^0\to\phi\mu^+\mu^-$ decays, however, the higher level of background contamination requires a tighter data selection and a more complex fit model. The $B_s^0\to f'_2(1525)\mu^+\mu^-$ decay is observed with a significance of $9\sigma$ based on Wilks's theorem, and the branching fraction is measured as $\mathcal{B}(B_s^0\to f'_2(1525)\mu^+\mu^-) = (1.57 \pm 0.19 \pm 0.06 \pm 0.06 \pm 0.08) \times 10^{-7}$ with (in order) the statistical uncertainty, the systematic uncertainty, the uncertainty due to the branching fraction of the normalisation mode, and the $q^2$-range extrapolation. This result is in agreement with the SM predictions.