Thunderstone Search Appliance Manual

Robots.txt

The Parametric Search Appliance comes with no robots.txt file. If you want to control how web walking robots visit your search pages you can upload a robots.txt file. This robots.txt file will apply to all pages returned bythe Parametric Search Appliance including admin, documentation, and searches whether accessed by HTTP or HTTPS. See http://www.robotstxt.org for the syntax of robots.txt files.

If you no longer wish to have a robots.txt check Delete.

Note: You may also control robots for searches on a profile by profile basis by using meta robots within the top HTML of the search settings or the custom XSL if you're using an XSL style sheet for search results. See http://www.robotstxt.org for the syntax of the meta robots tag.