Mild cognitive impairment (MCI) is a transitional state between the cognitive changes in normal aging and Alzheimer’s disease (AD), which induces abnormalities in specific brain regions. Previous studies showed that paired helical filaments Tau (PHF-Tau) protein is a potential pathogenic protein which may cause abnormal brain function and structure in MCI and AD patients. However, the understanding of the PHF-Tau protein network in MCI patients is limited. In this study, 225 subjects with PHF-Tau Positron Emission Tomography (PET) images were divided into four groups based on whether they carried Apolipoprotein E ε4 (APOE 4) or abnormal cerebrospinal fluid Total-Tau (CSF T-Tau). They are two important pathogenic factors that might cause cognitive function impairment. The four groups were: individuals harboring CSF T-Tau pathology but no APOE 4 (APOE 4−T+); APOE 4 carriers with normal CSF T-Tau (APOE 4+T−); APOE 4 carriers with abnormal CSF T-Tau (APOE 4+T+); and APOE 4 noncarriers with abnormal CSF T-Tau (APOE 4−T−). We explored the topological organization of PHF-Tau networks in these four groups and calculated five kinds of network properties: clustering coefficient, shortest path length, Q value of modularity, nodal centrality and degree. Our findings showed that compared with APOE 4−T− group, the other three groups showed different alterations in the clustering coefficient, shortest path length, Q value of modularity, nodal centrality and degree. Simultaneously, voxel-level analysis was conducted and the results showed that compared with APOE 4−T− group, the other three groups were found increased PHF-Tau distribution in some brain regions. For APOE 4+T+ group, positive correlation was found between the value of PHF-Tau distribution in altered regions and Functional Assessment Questionnaire (FAQ) score. Our results indicated that the effects of APOE 4 and abnormal CSF T-Tau may induce abnormalities of PHF-Tau protein and APOE 4 has a greater impact on PHF-Tau than abnormal CSF T-Tau. Our results may be particularly helpful in uncovering the pathophysiology underlying the cognitive dysfunction in MCI patients.