Abstract Background Patients who developed immune-related adverse events (irAEs) could benefit more from treatment with immune checkpoint inhibitors (ICIs) than those who did not develop irAEs. This study was designed to assess whether the occurrence of irAEs or their characteristics are correlated with survival in advanced patients treated with ICIs. Methods This retrospective cohort study enrolled a panel of cancer patients who received ICIs at a single institute. Kaplan‒Meier curves were generated to describe progression-free survival (PFS) and overall survival (OS) in patients with irAEs or specific irAE characteristics. Results A total of 238 patients were enrolled, 83 (34.9%) of whom developed at least one irAE. Overall, irAE development was associated with prolonged OS (not reached vs. 17.8 months, P < 0.001), PFS (8.7 vs. 4.8 months, P = 0.003), and an improved objective response rate (24.1% vs. 10.3%, P = 0.005). Furthermore, only skin or endocrine toxicities were associated with improved OS and PFS. On the basis of the results from organ-specific irAEs, the first development of skin or endocrine toxicities as protective irAEs rather than other irAEs was an independent indicator for predicting OS (P < 0.001) and PFS (P < 0.001). A protective irAE burden score based on organ-specific irAEs was further developed to show the significant protective effect of total irAEs on patient outcomes. Conclusions Not all irAEs are associated with prolonged survival. The identification of organ-specific irAEs is useful for stratifying patients who actually respond to and benefit from ICIs across different cancer types.