A Yb3+-doped borate glass system was examined for the structural and optical modifications after γ-irradiation. Among the studied 10BaO-20ZnO-20LiF-(50-x)B2O3-xYb2O3 (x = 0.1, 0.5, 0.7, and 1.0 mol%) glasses, the 10BaO-20ZnO-20LiF-49.9B2O3-0.1Yb2O3 glass showed the highest thermoluminescence intensity, trap density, and trap depth. The glass was irradiated with the optimum γ-dose of 1 kGy towards the analysis of radiation-induced defects. The amorphous nature was preserved before and after irradiation. The glass density slightly increased after irradiation. The structural rearrangement was evident from the Fourier transform infrared spectroscopy by the appearance and disappearance of some bonds after γ-irradiation. The transformation of [BO4] units into [BO3] units and non-bridging oxygens was deduced. The color of the glass darkened after irradiation and the optical absorption intensity enhanced between 250 and 700 nm. The optical bandgap reduced and Urbach energy increased upon γ-dose exposure. The electron spin resonance of the irradiated glass exhibited two signals at g = 2.0167 and g = 1.9938, corresponding to the non-bridging oxygen hole center and Boron E’-center, respectively.