Meta Robots Tag
The meta robots tag is an HTML tag in a page’s head that gives search engines page-level instructions, such as whether to index the page, follow its links or show a snippet in results. It is the most common way to keep individual pages out of search results.
How the tag looks
A basic example is:
<meta name="robots" content="noindex, follow">
The name attribute says which crawlers the rule applies to. robots means all search engines, while a specific name such as googlebot targets just one. The content attribute holds one or more directives separated by commas. If there is no meta robots tag, search engines assume the page can be indexed and its links followed.
Main directives
| Directive | What it does |
|---|---|
noindex | Keeps the page out of search results |
nofollow | Asks search engines not to follow links on the page |
none | Same as noindex, nofollow |
nosnippet | Stops a text or video snippet being shown |
max-snippet:[number] | Limits snippet length in characters |
max-image-preview:[setting] | Controls image preview size: none, standard or large |
noimageindex | Stops images on the page being indexed |
unavailable_after:[date] | Removes the page from results after a date |
Google has said that snippet controls such as nosnippet also apply to its AI features in Search, such as AI Overviews.
The noindex directive is covered in depth under noindex tag.
Meta robots versus robots.txt
These are often confused. Robots.txt controls crawling: whether a bot may fetch a URL at all. The meta robots tag controls indexing and display. Search engines must crawl a page to see its meta robots tag, so if you block a page in robots.txt, a noindex tag on it will never be read, and the URL can still appear in results if other sites link to it.
How to use it well
- Apply
noindexto pages with no search value, such as thank-you pages, internal search results and thin tag archives. - Keep those pages crawlable so search engines can see the instruction.
- Use an X-Robots-Tag header instead for PDFs and other non-HTML files.
- Check the tag on live pages with the URL Inspection tool, which shows whether indexing is allowed.
- Crawl the site regularly to catch accidental noindex tags on important pages.
Common mistakes
- Leaving noindex on after launch: staging sites are often noindexed, and the setting sometimes goes live with the new site, removing it from search.
- Using noindex instead of a canonical: for duplicate versions of a page, a canonical tag usually works better.
- Conflicting tags: if a page has several robots tags, Google follows the most restrictive.
- Hard-coded tags in themes: some themes and plugins add robots tags to whole page types, so check templates as well as individual pages.
- Expecting nofollow to hide pages: other links can still lead crawlers to them.
Further reading: Google’s robots meta tag specification.
Key Takeaways
- The meta robots tag gives page-level indexing and snippet instructions.
- Noindex keeps a page out of results, nofollow asks bots not to follow links.
- A page must be crawlable for search engines to see its meta robots tag.
- Use X-Robots-Tag for PDFs and other non-HTML files.
- Check for accidental noindex tags after every launch or migration.
Related Terms
Not sure how your own site measures up? Request a free SEO review, or keep learning in the SEO Encyclopedia.