We report a time-resolved measurement of the full transmission matrix (TM) of a short length of specialty annular-core few-mode fiber which guides 10 vector modes. We show how our method can isolate the fiber TM from “misalignment” contributions from optics upstream and downstream of the fiber. From measurements spanning two days, we extract the drift of the fiber TM. We show that drifts in the TM elements are mostly described as correlated phase variations rather than amplitude variations. We show that an empirical model of the fiber TM parametrized in one parameter can successfully account for the drift.