In engineering practice, layered rock masses often display obvious anisotropy while deforming and failing, and the failure mode directly impacts the engineering construction stability. In this study, the fracture failure load, fracture toughness, crack deflection angle, and failure mode of a layered rock mass under different fracture modes were analyzed by utilizing improved asymmetric semi-circular disc specimens. According to the constitutive model of transversely isotropic materials, the maximum tensile stress (MTS), maximum energy release rate (MERR), and maximum strain energy density (MSED) calculation formulas were modified, and the calculation formulas of the three prediction criteria under anisotropic materials were derived. The calculation results were compared with the experimental results. The results show that the fracture toughness and crack deflection angle were significantly affected by the weak bedding plane. As a result of applying the MTS criterion, the results are closer to the experimental results, providing a solid foundation for engineering deformation, failure, and fracture analyses.