Protecting wheat yield is a top priority in agricultural production, and one of the important measures to preserve yield is the control of wheat diseases. With the maturity of computer vision technology, more possibilities have been provided to achieve plant disease detection. In this study, we propose the position attention block, which can effectively extract the position information from the feature map and construct the attention map to improve the feature extraction ability of the model for the region of interest. For training, we use transfer learning to improve the training speed of the model. In the experiment, ResNet built on positional attention blocks achieves 96.4% accuracy, which is much higher compared to other comparable models. Afterward, we optimized the undesirable detection class and validated its generalization performance on an open-source dataset.