Esophageal squamous cell carcinoma (ESCC) is one of the deadliest solid malignancies and has a poor survival rate worldwide. In this study, we aimed to establish a tumor-infiltrating immune cell-based prognosis signature (IPS) to predict patients’ survival times and aid in the development of targeted therapies or immunotherapies. The abundances of 22 types of immune cells were determined by the CIBERSORT algorithm from ESCC patient gene expression data in the Gene Expression Omnibus (GEO) training set (n = 179) and The Cancer Genome Atlas (TCGA) validation set (n = 95). Then, the IPS was established by using the least absolute shrinkage and selection operator (LASSO) regression method. Kaplan-Meier analysis showed that patients with high IPS scores had significantly worse overall survival times than patients with low IPS scores in both the training set and the validation set (log-rank p = 0.001, and p = 0.050, respectively). Univariate and multivariate Cox regression analyses proved that the IPS was a robust prognostic factor for ESCC, independent of age, sex, tumor node metastasis (TNM) stage, pathology grade, and tumor location. In the mechanistic study, the epithelial-mesenchymal transition (EMT) process was identified by both gene set enrichment analysis (GSEA) and weighted correlation network analysis (WGCNA) as the underlying mechanism by which the IPS affects the prognosis of ESCC. After systematic correlation analyses, we found that M2 macrophages were the only cell type in the IPS significantly correlated with the EMT process. This relationship between M2 macrophage infiltration and the EMT phenotype was also confirmed by our preliminary immunochemistry (IHC) and multiplexed immunofluorescence study. In conclusion, we constructed an IPS that predicts the postoperative prognosis of ESCC patients and uncovered the critical role of M2 macrophages in the interplay between immune status and the EMT phenotype in ESCC.