Abstract Background Selecting the appropriate patients to receive immunotherapy (IO) remains a challenge due to the lack of optimal biomarkers. The presence of liver metastases has been implicated as a poor prognostic factor in patients with metastatic cancer. We investigated the association between sites of metastatic disease and clinical outcomes in patients receiving IO. Methods We conducted a retrospective review of 90 patients treated on IO-based phase 1 clinical trials at Winship Cancer Institute of Emory University between 2009 and 2017. Overall survival (OS) and progression-free survival (PFS) were measured from the first dose of IO to date of death or hospice referral and clinical or radiographic progression, respectively. Clinical benefit (CB) was defined as a best response of complete response (CR), partial response (PR), or stable disease (SD). Univariate analysis (UVA) and Multivariate analysis (MVA) were carried out using Cox proportional hazard model or logistic regression model. Covariates included age, whether IO is indicated for the patient’s histology, ECOG performance status, Royal Marsden Hospital (RMH) risk group, number of metastatic sites, and histology. Results The median age was 63 years and 53% of patients were men. The most common histologies were melanoma (33%) and gastrointestinal cancers (22%). Most patients (73.3%) had more than one site of distant metastasis. Sites of metastasis collected were lymph node (n = 58), liver (n = 40), lung (n = 37), bone (n = 24), and brain (n = 8). Most patients (80.7%) were RMH good risk. Most patients (n = 62) had received 2+ prior lines of systemic treatment before receiving IO on trial; 27 patients (30.0%) received prior ICB. Liver metastases were associated with significantly shorter OS (HR: 0.38, CI: 0.17–0.84, p = 0.017). Patients with liver metastasis also trended towards having shorter PFS (HR: 0.70, CI: 0.41–1.19, p = 0.188). The median OS was substantially longer for patients without liver metastases (21.9 vs. 8.1 months, p = 0.0048). Conclusions Liver metastases may be a poor prognostic factor in patients receiving IO on phase 1 clinical trials. The presence of liver metastases may warrant consideration in updated prognostic models if these findings are validated in a larger prospective cohort.