Thunderstone Search Appliance Manual

Walk Delay

Syntax: a decimal number from 0 to 10

Causes the Parametric Search Appliance to wait the specified number of seconds between page fetches. Normally set this to 0, and the Parametric Search Appliance will fetch and process pages as quickly as it can. Increase the Walk Delay if the web server cannot handle being hit rapidly. Increasing this value forces the walk to take at least the following number of seconds to complete: the Walk Delay number times the number of pages on the site.

Decimal numbers may be specified - 0.1 will cause it to walk no more than 10 pages per second, etc.

Note: Using a delay larger than 0 forces Threads (3.4.10) to 1 to avoid possible fetch timeouts. Thus a non-zero delay defeats the advantage of multiple threads. Also note that if Robots robots.txt is Y, then a Crawl-delay value in a site's robots.txt will override this setting.