Study and optimization on system architectures of Larbin

Abstract

Web crawler is an important part of the search engine,its performance will directly affect the accuracy and timeliness of the search engine.Larbin is an efficient and simple open source crawler with relatively perfect in functions.Several typical open-source crawler were firstly introduced and a multi-dimensional comparison was made among them.Then,the system architecture and working mechanism of Larbin were given in detail.Its short-comings in the program structure and process were pointed out,and improved programs were proposed.Experimen-tal results show that improved program is better in speed and performance.

Keywords

• search engine
• Web crawler
• Larbin
• open source
• optimization