# robots.txt for this web server # This file specifies rules for WWW robots to use when accessing # this server. See http://web.nexor.co.uk/mak/doc/robots/norobots.html # for details. User-agent: * Disallow: /cgi-bin/ # no access to dynamic stuff. Disallow: /htbin/ # ditto. Disallow: /temp/ # temp stuff.