# file: robots.txt , V1.0 2007/05/06 Created by YiGe&Wispom # www.gkcity.com # 按照robots.txt的标准写法,规定一些不允许爬虫爬的页面或目录。 # robots.txt 的写法参照 # Format is: # User-agent: # Disallow: | Baiduspider|iaskspider|Yahoo! |Googlebot|Msnbot|Alexibot|Sogou\+web\+spider|ia_archiver|sohu-search|soso|163 # ----------------------------------------------------------------------------- User-agent: * Crawl-delay: 100 Disallow: /bin/ Disallow: /App_Data/ Disallow: /Views/Security/ Disallow: /ScriptResource.axd Disallow: /WebResource.axd Sitemap: http://www.gkcity.com/sitemap.xml.gz Sitemap: http://www.gkcity.com/sitemap1.xml.gz Sitemap: http://www.gkcity.com/sitemap21.xml.gz Sitemap: http://www.gkcity.com/sitemap22.xml.gz Sitemap: http://www.gkcity.com/sitemap3.xml.gz