Search for new interactions in the top quark sector

The top quark is the heaviest elementary particle discovered so far and the study of its unique properties provides not only an essential test of the Standard Model of Particle Physics, but also an important window into physics beyond it. This thesis presents a search for Flavour Changing Neutral Currents processes through the top quark interactions with a neutral $Z$ boson and a light quark, that can be an up or charm quark. The analysis uses a 139 fb$^{-1}$ dataset of proton-proton collisions collected at a center-of-mass of $\sqrt{s}$ = 13 TeV with the ATLAS detector, at the Large Hadron Collider between 2015 and 2018. The presented analysis focus on the $tZq$ vertex interactions with a final state composed by three leptons, resulting from the leptonic decays of the top quark and the $Z$ boson. In order to improve the sensitivity of the search, events from both production and decay modes are considered, being split in distinct signal regions according with the number of jets. Methods based on multivariate techniques were developed and applied to dedicated signal regions to discriminate $tZq$ events against background events, which are dominated by top quark pair production in association with a $Z$ boson as well as diboson production. All the analysis' regions were combined in a statistic model using a profile likelihood fit to constrain the background predictions and reduce the systematic uncertainties. Since no evidence for a signal was found, upper limits on the cross-section and branching ratio of these processes at 95\% confidence level were evaluated. The observed (expected) limit on the branching ratio of $t\to uZ$ is $6.16$ ($4.88$) $\times10^{-5}$, while the observed (expected) limit on the branching ratio of $t\to cZ$ is $13.02$ ($10.76$) $\times10^{-5}$. Additionally, observed (expected) limits excluding cross-sections of $\sigma(pp\to tZ)$ above $37$ ($29$) fb and $118$ ($96$) fb were obtained for the $tZu$ and $tZc$ processes. Through a phenomenological study, the search for a new pseudo-scalar particle $S$ has also been investigated focusing on several leptonic final states. The sensitivity study of these dedicated analyses allowed to conclude that new physics scales as large as $\Lambda \sim$ 90 TeV can be probed with an integrated luminosity of 150 fb$^{-1}$.