Infertility is a global health problem, affecting approximately one sixth of the population of reproductive age, and in nearly 30% of cases the etiology of male infertility remains unexplained. In recent years, increasing attention has been paid to the role of inflammatory processes associated with infections, autoimmune diseases, as well as environmental and lifestyle factors. Testis inflammation and the accompanying oxidative stress disrupt spermatogenesis and androgen biosynthesis. Understanding the impact of inflammatory processes on male reproductive function requires consideration of the unique organization and regulation of the immune response in the testis. The mammalian testis is an immunologically privileged organ, wherein specialized cellular and molecular mechanisms control immune recognition of germ cell antigens and prevent the initiation of inflammatory responses. Regulation of testicular immune privilege is based on systemic immune tolerance and site specific active immunosuppression. According to current knowledge it involves the blood–testis barrier, local immunosuppressive mechanisms, resident immune cells, and testis-specific mechanisms and cells. This review aims to explore the role of Sertoli cells, Leydig cells, and peritubular myoid cells in maintaining testis immune privilege, taking into account current concepts and the latest research findings. Factors produced by these cells and their effects on immune system cells are discussed. The impact of testicular inflammation on the immunosuppressive properties of Sertoli, Leydig, and myoid cells is also briefly presented.