Nowadays, nanotechnology has become a very attractive subject in Enhanced Oil Recovery (EOR) researches. In the current study, a carbonate system has been selected and first the effects of nanoparticles on the rock and fluid properties have been experimentally investigated and then the simulation and numerical modeling of the nanofluid injection for enhanced oil recovery process have been studied. After nanofluid treatment, experimental results have shown wettability alteration. A two-phase flow mathematical model and a numerical simulator considering wettability alteration have been developed. The numerical simulation results show that wettability alteration from oil-wet to water-wet due to presence of nanoparticles can lead to 8–10% increase in recovery factor in comparison with normal water flooding. Different sensitivity analyses and injection scenarios have been considered and assessed. Using numerical modeling, wettability alteration process and formation damage caused by entrainment and entrapment of nanoparticles in porous media have been proved. Finally, the net rate of nanoparticles’ loss in porous media has been investigated.