Study of heavy flavour decay muons at forward rapidity in proton-proton and heavy-ion collisions at LHC energies

In high energy collision experiment, the heavy quarks (charm and beauty) are produced mainly in the initial stage of the collisions via hard partonic scattering processes. The study of heavy flavour production in proton-proton (pp) collisions at LHC energy regime provides an important test for perturbative Quantum Chromodynamics (pQCD). Furthermore, the measurement of heavy flavour production in pp collisions serves as a necessary baseline for the same measurement in proton-nucleus (p-A) and nucleus-nucleus (A-A) collisions. The productions of heavy flavour hadrons are measured by ALICE (A Large Ion Collider Experiment) detector at LHC via their semileptonic (electronic or muonic) decay. The measurement of single muons through semi-muonic decay of heavy flavour hadrons is done using the Muon Spectrometer (MS) of ALICE. ALICE has measured the heavy flavour hadron decay muon (HFM) production cross-section at forward rapidity ($-4<\eta<-2.5$) in pp collisions at $\sqrt{s} = $ 2.76 TeV, 5.02 TeV and 7 TeV respectively. In this thesis, the HFM production cross-section is measured at the highest available energy $\sqrt{s} = $ 13 TeV of LHC during Run 2 using MS of ALICE detector with an extended transverse momentum range $2<p_{T}<30$ GeV/$\it{c}$. This new measurement of the production of HFM is compared with the theoretical FONLL (Fixed-Order + Next-to-Leading Logarithms) calculations and found to be in good agreement almost over the whole $p_{T}$ range within the theoretical uncertainties.\\ After the discovery of the features of HI collisions such as ridge like effect and strangeness enhancement in high multiplicity pp events at LHC, it is of very interest to investigate the collective effects even in small colliding systems. A phenomenological work has been also presented to study the heavy flavour decay muon (HFM) and charged particles production in proton-proton(pp) and heavy-ion(A-A) collisions using Angantyr model at LHC energies. The Angantyr model extrapolates the pp dynamics using the PYTHIA8 machinery into a HI collisions and does not assume any thermalised medium (QGP) to be produced in HI collisions. One of the main motivation of this phenomenological study is to see how well such extrapolation of pp dynamics can reproduce the experimental data for HI collisions. The study using the Angantyr model could be considered as a baseline for the understanding of non-collective background to the observables sensitive to collective behaviour.\\ In the upgrade program of ALICE for Run 3, a new detector called Muon Forward Tracker (MFT) is installed upstream of the front absorber of MS. The addition of MFT is expected to improve the vertexing capabilities of muon tracks to the existing MS. The basic detection elements of MFT are silicon pixel sensors which are integrated on a mechanical structure called ladder. The description of the procedures of Quality Assurance test for qualification of MFT-ladders has been presented.