Effect of anisotropy in compression is studied on hot rolling of AZ31 magnesium alloy with a three-dimensional constitutive model based on the quadratic Hill48 yield criterion and nonassociated flow rule (non-AFR). The constitutive model is characterized by compressive tests of AZ31 billets since plastic deformations of materials are mostly caused by compression during rolling processes. The characterized plasticity model is implemented into ABAQUS/Explicit as a user-defined material subroutine (VUMAT) based on semi-implicit backward Euler's method. The subroutine is employed to simulate square-bar rolling processes. The simulation results are compared with rolled specimens and those predicted by the von Mises and the Hill48 yield function under AFR. Moreover, strip rolling is also simulated for AZ31 with the Hill48 yield function under non-AFR. The strip rolling simulation demonstrates that the lateral spread generated by the non-AFR model is in good agreement with experimental data. These comparisons between simulation and experiments validate that the proposed Hill48 yield function under non-AFR provides satisfactory description of plastic deformation behavior in hot rolling for AZ31 alloys in case that the anisotropic parameters in the Hill48 yield function and the non-associated flow rule are calibrated by the compressive experimental results.