The self-reference effect (SRE) is defined as better recall or recognition performance when the memorized materials refer to the self. Recently, a number of neuroimaging studies using self-referential and other-referential tasks have reported that self- and other-referential judgments basically show greater activation in common brain regions, specifically in the medial prefrontal cortex (MPFC) when compared with nonmentalizing judgments, but that a ventral-to-dorsal gradient in MPFC emerges from a direct comparison between self- and other-judgments. However, most of these previous studies could not provide an adequate explanation for the neural basis of SRE because they did not directly compare brain activation for recognition/recall of the words referenced to the self with another person. Here, we used an event-related functional magnetic resonance imaging (fMRI) that measured brain activity during processing of references to the self and another, and for recognition of self and other referenced words. Results from the fMRI evaluation task indicated greater activation in ventromedial prefrontal cortex (VMPFC) in the self-referential condition. While in the recognition task, VMPFC, posterior cingulate cortex and bilateral angular gyrus showed greater activation when participants correctly recognized self-referenced words versus other-referenced words. These data provide evidence that the self-referenced words evoked greater activation in the self-related region (VMPFC) and memory-related regions (PCC and angular gyrus) relative to another person in the retrieval phase, and that the words remained as a stronger memory trace that supports recognition.