Web Crawler

It is the software agent that crawl pages from the web and feed the search engine. searchbox implements a highly powerful web crawler that also can gather information from many other types of sources with protocols like FTP, NNTP, Gopher, RSS, ODBC, WebDav, etc. For more information about the web crawler implemented into searchbox please see the searchbox essentials section of the user manual.