Abstract The 4D scanning transmission electron microscopy (STEM) method maps the structure and functionality of solids on the atomic scale, yielding information-rich data sets describing the interatomic electric and magnetic fields, structural and electronic order parameters, and other symmetry breaking distortions. A critical bottleneck is the dearth of analytical tools that can reduce complex 4D-STEM data to physically relevant descriptors. We propose an approach for the systematic exploration of 4D-STEM data using rotationally invariant variational autoencoders (rrVAE), which disentangle the general rotation of the object from other latent representations. The implementation of purely rotational rrVAE is discussed as are applications to simulated data for graphene and zincblende structures. The rrVAE analysis of experimental 4D-STEM data of defects in graphene is illustrated and compared to the classical center-of-mass analysis. This approach is universal for probing symmetry-breaking phenomena in complex systems and can be implemented for a broad range of diffraction methods.