According to SEO expert from Poush, search engine spider crawl and index web pages of a site which are stored in database. The rankings and relevancy of the collected pages are determined by various algorithms used by search engine spiders. As the index pages are more or less uniform while calculating rankings using algorithms, SEO experts apply proper SEO activities knowing the interest of spiders.

While crawling a web page, the spider does not read Flash and JavaScript as the human does. Frames, Flash and JavaScript in a website are good for the users but are ignored by search engines. The major problem is that the keywords buried in the flash displays are not readable by the spiders. Having an image with text is nothing to a spider and unless an ALT attribute is specified for images in web portal it is difficult for spider to crawl these images. SEO experts do not use many keywords for an image as it will result into keyword stuffing. It is necessary to check the pages showing “403 forbidden” or “404 page not found” whenever the spider crawls the website. This will make other web pages to be easily indexed and improves efficiency of web portal. Initially keywords related to the business are selected for ranking in search engines such as Google, Yahoo and Bing. After selection of keywords and manipulation, these are added in the content of each page which helps the crawler to identify the words that are needed to be ranked in search engines for the business.
Meta keywords and meta description found in the code of web pages are important for search engine spider. SEO experts apply on page and off page optimization such as meta tagging, link building, URL structuring as well as social bookmarking to a web portal. If the quality links to the website increases then the crawler gives more preference to its web pages in Google, Yahoo and Bing. One of the things SEO experts must not forget is sitemap of website which helps the spider to know the web pages with uniform resource locators (URLs). Updating new content makes the crawler active and stay longer on the web pages. Google webmaster tools are used by the experts to identify the performance of the crawler on the web portal. Search engine spiders play a crucial role while ranking a site for selected keywords.