User-agent: ia_archiver Disallow: / User-agent: archive.org_bot Disallow: / User-agent: Googlebot User-agent: AdsBot-Google Disallow: / #Example of how to block all crawlers User-agent: * Disallow: /