User-agent: Mediapartners-Google* Disallow: User-Agent: Googlebot #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /stories/*/comments$ Disallow: /stories/*/full_content$ Disallow: /softlaunch Disallow: /history User-Agent: Cogger #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history # Alexa Archver, allow them User-Agent: ia_archiver Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history # Internet Archives open source crawler # Has gone nuts on us before. User-Agent: os-heritrix Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-agent: YahooFeedSeeker Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-agent: BlogPulse Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-agent: kinjabot Disallow: /comment Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: Teoma Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: Sphere Scout Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: msnbot Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: archive.org_bot Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /softlaunch Disallow: /history # topsy.com's bot User-Agent: Butterfly Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: Yandex Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: ScoutJet Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: cdlwas_bot Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: dk-bot Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: bingbot Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /stories/*/comments$ Disallow: /stories/*/full_content$ Disallow: /news Disallow: /softlaunch Disallow: /history Crawl-delay: 1 User-Agent: special_archiver Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: Diffbot Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-Agent: Crowsnest Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /softlaunch Disallow: /history User-agent: Twitterbot Disallow: User-Agent: * Disallow: /