Viscoelasticity of the soft tissue is an important mechanical factor for disease diagnosis, biomaterials testing and fabrication. Here, we present a real-time and high-resolution viscoelastic response-optical coherence elastography (VisR-OCE) method based on acoustic radiation force (ARF) excitation and optical coherence tomography (OCT) imaging. The relationship between displacements induced by two sequential ARF loading—unloading and the relaxation time constant of the soft tissue—is established for the Kelvin-Voigt material. Through numerical simulation, the optimal experimental parameters are determined, and the influences of material parameters are evaluated. Virtual experimental results show that there is less than 4% fluctuation in the relaxation time constant values obtained when various Young’s modulus and Poisson’s ratios were given for simulation. The accuracy of the VisR-OCE method was validated by comparing with the tensile test. The relaxation time constant of phantoms measured by VisR-OCE differs from the tensile test result by about 3%. The proposed VisR-OCE method may provide an effective tool for quick and nondestructive viscosity testing of biological tissues.