When a search engine want's to index the web, it uses a program called a spider or robot. The search engine sends out its spider to crawl the web to find new pages to index so it can build a database from the information it discovers. The spider indexes content such as HTML, images PDF files, or any other file found on the web. The terms spider, spidering, crawl, or crawling are commonly used indexing the content it finds.
Wednesday, 20 November 2013
Understanding a Spider's Role in the Web
When a search engine want's to index the web, it uses a program called a spider or robot. The search engine sends out its spider to crawl the web to find new pages to index so it can build a database from the information it discovers. The spider indexes content such as HTML, images PDF files, or any other file found on the web. The terms spider, spidering, crawl, or crawling are commonly used indexing the content it finds.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment