Background: Patients with primary antibody deficiencies are at risk in the current COVID-19 pandemic due to their impaired response to infection and vaccination. Specifically, patients with common variable immunodeficiency (CVID) generated poor spike-specific antibody and T cell responses after immunization. Methods: Thirty-four CVID convalescent patients after SARS-CoV-2 infection, 38 CVID patients immunized with two doses of the BNT162b2 vaccine, and 20 SARS-CoV-2 CVID convalescents later and immunized with BNT162b2 were analyzed for the anti-spike IgG production and the generation of spike-specific memory B cells and T cells. Results: Spike-specific IgG was induced more frequently after infection than after vaccination (82% vs. 34%). The antibody response was boosted in convalescents by vaccination. Although immunized patients generated atypical memory B cells possibly by extra-follicular or incomplete germinal center reactions, convalescents responded to infection by generating spike-specific memory B cells that were improved by the subsequent immunization. Poor spike-specific T cell responses were measured independently from the immunological challenge. Conclusions: SARS-CoV-2 infection primed a more efficient classical memory B cell response, whereas the BNT162b2 vaccine induced non-canonical B cell responses in CVID. Natural infection responses were boosted by subsequent immunization, suggesting the possibility to further stimulate the immune response by additional vaccine doses in CVID.