Search for narrow low-mass resonances in the four-muon final state with the ATLAS detector at the LHC

A search for new low-mass resonances in the four-muon final state in the four-muon invariant mass range from $10-50$~GeV is performed using $pp$ collision data recorded with the ATLAS detector at the Large Hadron Collider. The data correspond to an integrated luminosity of 20.3~fb$^{-1}$ at a centre-of-mass energy ($\sqrt{s}$) of 8~TeV collected in 2012, and 51.5~fb$^{-1}$ and 58.5~fb$^{-1}$ collected at $\sqrt{s}=13$~TeV in 2015--2017 and 2018, respectively. An excess at 18~GeV in the invariant mass spectrum of the four-muon system of the selected $\Upsilon(1\mathrm{S})+\mu^+\mu^-\to\mu^+\mu^-\,\mu^+\mu^-$ final state is observed in the 8~TeV data. Although not blinded, typical selection criteria are used and variations studied to explore the stability of the excess, resulting in local significances of the excess of between be $3.6\,\sigma$ and $6.3\,\sigma$ corresponding to global significances of $1.9\,\sigma$ to $5.4\,\sigma$. A baseline choice of selection cuts defined for the analysis at 8~TeV was defined in the 8~TeV data to be further probed at Run 2. Subsequent analysis of 13~TeV data finds a smaller excess at the same four-muon invariant mass with a significance of $1.9\,\sigma$ in early 13~TeV data runs, while data collected with more restrictive trigger selections in the 2018 dataset finds no evidence of a signal. LHCb and CMS have observed no excess at this mass. Signal MC and di-$\Upsilon(1\mathrm{S})$ production studies in data are used to quantify performance across the three data-taking periods, finding reductions in expected sensitivity in the 13~TeV dataset compared to that at 8~TeV. The lack of observation of a signal in the 13~TeV is assessed to be in tension with the structure observed at an invariant mass of 18~GeV in the 8 TeV dataset, up to a maximum of $2.7\sigma$. We subsequently set and report limits on the total production cross section times branching fraction to a four-muon final state for a new state at a mass of 18~GeV for a variety of models.