Femtoscopic analysis of hadron-hadron correlations in ultrarelativistic collisions of protons and heavy-ions registered by ALICE at the LHC

One of the most powerful methods developed to probe the properties of the Quark-Gluon Plasma (QGP) is the technique of two-particle correlations in momentum space, called \emph{femtoscopy}. Femtoscopy gives the unique possibility to measure the space-time evolution of the system created in particle collisions. It is capable of measuring space scales of the order of femtometers ($10^{-15}$~m; the size of a nucleon), as well as times of the order of $10^{-23}$~s. In heavy-ion collisions it provides insight into the collective effects exhibited by the created bulk strongly coupled matter. Intriguing results from the analysis of the p--Pb collisions at the Large Hadron Collider (LHC) suggest that collective properties could also develop in small systems after all. First studies of the p--Pb system, which was initially expected to serve as a control measurement, assuming no formation of a QGP state, show that particularly at the extreme energies at the LHC, more complex physics mechanisms maybe involved, interesting on their own right. Therefore, systematic femtoscopic measurements of pp, p--Pb, and Pb--Pb systems are expected to provide crucial experimental input that can advance our understanding of the QGP state of matter and provide further constraints on its properties and characteristics. Typically, pion-pion or kaon-kaon correlations are studied in order to determine the source size and its evolution in time. However, the femtoscopic formalism is not restricted to light mesons only. Other particles, in particular baryons, are also studied. For the study of baryon-baryon correlations the femtoscopic formalism can be employed in a novel way allowing us to extract the strong interaction parameters, which are known only for a few of the lightest baryon systems, like proton-proton, proton-neutron, proton-deuteron, etc. One of the surprising results from Pb--Pb collisions at the LHC are the yields of protons and lambda hyperons, which are lower than predicted by extrapolations from lower energies. It is suggested that one aspect of the strong interaction, the annihilation of different baryon-antibaryon pairs, is responsible for these observations. Femtoscopy can be employed to measure the baryon-antibaryon interactions which then can be used to explain this effect. This thesis presents the results of two-pion femtoscopy in p--Pb collisions at $\sqrt{s_{\rm NN}}=5.02$~TeV and preliminary results of baryon femtoscopy in Pb--Pb collisions at $\sqrt{s_{\rm NN}}=2.76$~TeV delivered by the LHC and registered by the ALICE detector. For the p--Pb system there are, in general, predictions of the source sizes originating from two possible physics mechanisms. The potential existence of a collective phase in high-multipli- city p--Pb collisions is predicted to increase the measured femtoscopic radii by a factor of $1.5$--$2$ with respect to the pp collisions at similar multiplicity. On the other hand, within a Color Glass Condensate (CGC) initial-state model, without a hydrodynamic phase, similar source sizes in both p--Pb and pp collisions are predicted. To verify these scenarios the three-dimensional pion femtoscopic radii were measured for the first time in four multiplicity and seven pair transverse momentum $k_{\rm T}$ ranges. Similarly to A--A and high multiplicity pp~collisions, the radii decrease with $k_{\rm T}$ in all cases. They also increase with event multiplicity. At low multiplicity they are comparable to pp~values, while at higher multiplicities and low $k_{\rm T}$ they are larger by 10--20\%. However, the p--Pb radii do not reach the values observed in A--A collisions at lower energies. The high multiplicity p--Pb data were compared to predictions from two hydrodynamic models. They provide larger values of the $R_{\mathrm{out}}$ and $R_{\mathrm{long}}$ parameters; however, the introduction of smaller initial size of the system brings calculations closer to the experimental data. In particular, the $R_{\mathrm{side}}$ parameter and the slope of the $k_{\rm T}$ dependence of the radii are in reasonable agreement. Nevertheless, the observed differences of 10--20\% between high multiplicity pp and p--Pb collisions do not exclude the CGC scenario. In Pb--Pb collisions the correlations of protons with lambda hyperons were measured. The correlation functions for different systems (baryon-baryon: p$\Lambda$, $\mathrm{\overline{p}}\overline{\Lambda}$, as well as baryon-antibaryon: p$\overline{\Lambda}$, $\mathrm{\overline{p}}\Lambda$) were obtained for five centrality ranges. Baryon-antibaryon correlations were qualitatively compared to theoretical expectations calculated using the Lednicky \& Lyuboshitz analytical model with and without the annihilation process. The qualitative comparison revealed the presence of a wide anticorrelation that can be interpreted as a significant contribution of the baryon-antibaryon annihilation process. This result is the starting point for the future measurements of the interaction cross sections in these systems, which will be used to propose an explanation for the baryon yields at the LHC. Both measurements presented in this thesis, the pion femtoscopy in p--Pb collisions and baryon femtoscopy in Pb--Pb collisions, address different aspects of QGP and contribute to its understanding and characterization. Combined with other measurements, they provide further insight into the processes which occur in strongly coupled matter at extreme energy density.