Summary: Following viral infection, the human immune system generates CD8+ T cell responses to virus antigens that differ in specificity, abundance, and phenotype. A characterization of virus-specific T cell responses allows one to assess infection history and to understand its contribution to protective immunity. Here, we perform in-depth profiling of CD8+ T cells binding to CMV-, EBV-, influenza-, and SARS-CoV-2-derived antigens in peripheral blood samples from 114 healthy donors and 55 cancer patients using high-dimensional mass cytometry and single-cell RNA sequencing. We analyze over 500 antigen-specific T cell responses across six different HLA alleles and observed unique phenotypes of T cells specific for antigens from different virus categories. Using machine learning, we extract phenotypic signatures of antigen-specific T cells, predict virus specificity for bulk CD8+ T cells, and validate these predictions, suggesting that machine learning can be used to accurately predict antigen specificity from T cell phenotypes.