Anomaly detection search for new resonances decaying into a Higgs boson and a generic new particle $X$ in hadronic final states using $\sqrt{s}$ = 13 TeV $pp$ collisions with the ATLAS detector

A search is presented for a heavy resonance $Y$ decaying into a Standard Model Higgs boson $H$ and a new particle $X$ in a fully hadronic final state. The full Large Hadron Collider Run 2 $\sqrt{s} = 13$ TeV $pp$ dataset is used, collected by the ATLAS detector from 2015 to 2018 and corresponding to an integrated luminosity of 139 fb$^{-1}$. The search targets the high $Y$ mass regime, where the $H$ and $X$ have a significant Lorentz boost. A novel anomaly detection signal region is implemented based on a jet-level score for signal model-independent tagging of the boosted $X$, representing the first application of fully unsupervised machine learning to an ATLAS analysis. Two additional signal regions are implemented to target a benchmark $X$ decay to two quarks, which cover topologies where the $X$ is reconstructed as either a single large-radius jet or two small-radius jets. The Higgs is assumed to decay to $b\bar{b}$ and a dedicated neural net-based tagger is used for sensitivity to the boosted heavy flavor topology. No significant excess of data is observed over the expected background, and the results are interpreted in upper limits at 95% confidence level on the production cross section $\sigma(pp \rightarrow Y \rightarrow XH \rightarrow q\bar{q}b\bar{b}$) for signals with $m_Y$ between 1.5 and 6 TeV and $m_X$ between 65 and 3000 GeV.