Indexing examples
Mike Banks Valentine has an article based on his experience on how spiders behave when indexing. For example he concludes:
1) Google crawls 250 pages on first discovery of links to site. Then they don’t return until they find more links and crawl slowly. Google has failed to index new domain for 60 days.
2) Yahoo looks for errors pages and once they find bad links will crawl them ceaselessly until you tell them to stop it. Then won’t crawl at all for weeks until crawling heavily one day and lightly the next in random fashion.
Even though I don’t agree with it’s conclusions on Google (see this example, the part on Yahoo it’s interesting.
The complete results of his analysis is here . What I see that he ommited is what submission steps he took if any. As far as I know no one will spider your domain out of the blue ![]()