# Allow and Disallow all crawlers access to certain pages. User-Agent: * Allow: / Allow: /shop Allow: /fff Disallow: /error Disallow: /shop/web/basket.do Disallow: /manager Disallow: /manager_dir