According to SEO expert from Poush, search engine spider crawl and index web pages of a site which are stored in database. The rankings and relevancy of the collected pages are determined by various algorithms used by search engine spiders. As the index pages are more or less uniform while calculating rankings using algorithms, SEO experts apply proper SEO activities knowing the interest of spiders.
Meta keywords and meta description found in the code of web pages are important for search engine spider. SEO experts apply on page and off page optimization such as meta tagging, link building, URL structuring as well as social bookmarking to a web portal. If the quality links to the website increases then the crawler gives more preference to its web pages in Google, Yahoo and Bing. One of the things SEO experts must not forget is sitemap of website which helps the spider to know the web pages with uniform resource locators (URLs). Updating new content makes the crawler active and stay longer on the web pages. Google webmaster tools are used by the experts to identify the performance of the crawler on the web portal. Search engine spiders play a crucial role while ranking a site for selected keywords.