X-Robots-Tag

The X-Robots-Tag is an HTTP response header that gives search engines indexing and snippet instructions, using the same directives as the meta robots tag. Because it sits in the server response rather than the HTML, it works for any file type, including PDFs, images and videos.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

How it works

When a server sends a file, it includes headers with information about the response. Adding an X-Robots-Tag header looks like this:

X-Robots-Tag: noindex

Search engines read the header when they fetch the file and apply the rule. You can combine directives, such as X-Robots-Tag: noindex, nofollow, and target a specific crawler, such as X-Robots-Tag: googlebot: noindex. The directives are the same as those for the meta robots tag, including noindex, nofollow, nosnippet and unavailable_after.

When to use X-Robots-Tag

  • Non-HTML files: PDFs, Word documents, spreadsheets and images have no HTML head, so a meta tag is impossible. The header is the only page-level way to noindex them.
  • Rules for many files at once: a single server rule can noindex every file in a folder or of a certain type.
  • Whole environments: adding noindex to every response on a staging server helps keep staging sites out of search, though password protection is safer.
  • Templates you cannot edit: if the CMS does not let you change the HTML head, a header set at server or CDN level can do the job.

How to set it up

  1. On Apache, add a Header set X-Robots-Tag "noindex" rule in the server config or .htaccess, often inside a block matching file types such as PDF.
  2. On Nginx, use add_header X-Robots-Tag "noindex"; inside the relevant location block.
  3. Some CDNs and hosting panels let you add response headers without editing server files.
  4. Check the result in browser developer tools under the Network tab, or with the URL Inspection tool in Search Console.

X-Robots-Tag and robots.txt

As with the meta robots tag, search engines must be able to fetch a file to see its header. If you block a PDF in robots.txt, Google never sees the noindex instruction, and the URL can still be indexed from links.

Leave the file crawlable and use the header to keep it out of the index. The directive works the same way as a noindex tag once read.

Common mistakes

  • Accidental site-wide noindex: a header rule copied from staging to live can quietly deindex the whole site. Because it is invisible in the page source, it is easy to miss.
  • Overly broad patterns: a rule meant for one folder can match far more URLs than intended.
  • Forgetting important PDFs: some PDFs, such as guides, may deserve to rank. Only noindex those with no search value.
  • Not auditing headers: run a crawler that reports X-Robots-Tag values, especially after a website migration.

Further reading: Google’s robots meta tag and X-Robots-Tag specification.

Key Takeaways

  • X-Robots-Tag sends robots directives in the HTTP response header.
  • It works for any file type, including PDFs and images.
  • It uses the same directives as the meta robots tag.
  • Files must stay crawlable for search engines to read the header.
  • Audit headers after launches, as accidental noindex rules are easy to miss.

Related Terms

Wondering what to fix first on your site? Get a free check of your site’s SEO, or keep learning in the SEO Encyclopedia.