A Web-Based Crawler Using Bee Swarm Intelligent Algorithm

Abstract

One of the major challenges with most swarm intelligence algorithms in multimodal optimization is premature convergence. Search engines are using web spiders to crawl the web in order to collect copies of the web sites for their databases. These spiders usually use the technique of breadth first search which is non- guided or blind that depends on visiting all links of any web site one- by-one. We have studied various swarm intelligence algorithm with the search engine to enrich the literature. In this work, we develop a  Web-based Crawler Using Bee Swarm Intelligent Algorithm. The methodology used is Structured System Analysis and Design Methodology(SSADM) in this approach. We implemented with PHP programming language and MySQL as database. The expected results show links into different module in a closed system, which appear to be more productive, fast and more flexible when compared with existing web crawlers. This work could be beneficial to the government, research communities and any other organisations that deals with information retrieval.