Searches for exclusive Higgs and Z boson decays into a vector quarkonium state and a photon with the ATLAS experiment

In the Standard Model (SM) the mass generation of fermions is implemented through Yukawa couplings to the Higgs boson. Experimental evidence exists for the Higgs boson couplings to second and third generation leptons through its decay to muon and tau pairs, but for quarks direct evidence exists only for the third-generation couplings, and direct searches for inclusive decays of the Higgs boson to lighter generations are challenging due to large QCD backgrounds at the LHC. With their distinct experimental signature, radiative decays of the Higgs boson to a meson and a photon, complemented by searches for analogous decays of the Z boson, offer an alternative probe of quark Yukawa couplings. Moreover, these decays provide an opportunity to investigate physics beyond-the-SM, as many such theories predict branching fractions significantly modified from the SM expectation. The rare decays of the Higgs boson in the charmonium sector, to a J/psi or psi(2S) state and a photon, provide an opportunity to access the not-yet observed charm-quark Yukawa coupling; the rare decays of the Higgs boson in the bottomonium sector, to an Upsilon(1S,2S,3S) state and a photon, can provide information about the real and imaginary parts of the bottom-quark Yukawa coupling, and are particularly sensitive to deviations from the SM. The corresponding Z boson decays to the same final state can provide a useful benchmark channel for the Higgs boson decays, but also offer an opportunity to test the QCD factorisation approach. Upper limits on branching ratios of the rare decays of the Higgs and Z boson to a vector quarkonium state and a photon were set at the ATLAS experiment using 36.1 fb$^{−1}$ of ATLAS data at $\sqrt{s}$ = 13 TeV, corresponding to the 2015-2016 dataset. This poster will showcase the results of the latest search, which uses the full 139 fb$^{−1}$ ATLAS dataset from 2015-2018. This search targets the quarkonium decays to dimuons and uses dedicated single photon plus muon triggers. With the increased statistics and new approach to modelling the resonant background, the limits on the branching ratios of each decay channel are improved by a factor of approximately two compared to the previous result. Combined limits are also set on the Higgs and Z boson decays to either J/psi or psi(2S) and a photon, and to any of the Upsilon(1S,2S,3S) states and a photon.