Severe drug allergy affects patient hesitancy to new treatments, posing unprecedented challenges to anti-SARS-CoV-2 vaccination campaigns. We aimed to analyze the psychological profile of vaccinees with a history of severe allergy in comparison to subjects with a milder allergy history. Patients attending a dedicated vaccination setting were administered an anonymized questionnaire including clinical data and the State-Trait Anxiety Inventory (STAI) scale (score range 20–80). Patients were also asked whether being in a protected setting affected their attitude toward vaccination. Data are expressed as median (interquartile range). We enrolled 116 patients (78% women), of whom 79% had a history of drug anaphylaxis. The median state anxiety score was 36.5 (30–47.2), while the trait anxiety score was 37 (32–48). State anxiety was higher in those with severe than mild allergy [39 (32–50) vs. 30 (25–37); p < 0.001], with the highest score found in a patient with previous drug anaphylaxis (42.5 [32–51.7]). More than 50% of patients reported that being in a protected setting had lowered their anxiety. Severe allergy is associated with a higher burden of situational anxiety in the setting of vaccination without affecting patient constitutional (trait) levels of anxiety. Vaccination in dedicated facilities might overcome issues related to hesitancy and improve patients’ quality of life.