Abstract Background Language deficits frequently occur during the prodromal stages of Alzheimer’s disease (AD). However, the characteristics of linguistic impairment and its underlying mechanism(s) remain to be explored for the early diagnosis of AD. Methods The percentage of silence duration (PSD) of 324 subjects was analyzed, including patients with AD, amnestic mild cognitive impairment (aMCI), and normal controls (NC) recruited from the China multi-center cohort, and the diagnostic efficiency was replicated from the Pitt center cohort. Furthermore, the specific language network involved in the fragmented speech was analyzed using task-based functional magnetic resonance. Results In the China cohort, PSD increased significantly in aMCI and AD patients. The area under the curve of the receiver operating characteristic curves is 0.74, 0.84, and 0.80 in the classification of NC/aMCI, NC/AD, and NC/aMCI+AD. In the Pitt center cohort, PSD was verified as a reliable diagnosis biomarker to differentiate mild AD patients from NC. Next, in response to fluency tasks, clusters in the bilateral inferior frontal gyrus, precentral gyrus, left inferior temporal gyrus, and inferior parietal lobule deactivated markedly in the aMCI/AD group (cluster-level P < 0.05, family-wise error (FWE) corrected). In the patient group (AD+aMCI), higher activation level of the right pars triangularis was associated with higher PSD in in both semantic and phonemic tasks. Conclusions PSD is a reliable diagnostic biomarker for the early stage of AD and aMCI. At as early as aMCI phase, the brain response to fluency tasks was inhibited markedly, partly explaining why PSD was elevated simultaneously.