Screaming Frog

Screaming Frog usually refers to the Screaming Frog SEO Spider, a desktop website crawler made by a UK search marketing company of the same name. It crawls a site much like a search engine bot, then lists every URL with details such as status codes, titles, meta descriptions, headings, canonicals and links, making it a standard tool for technical audits.

Written by Paul Hoda, SEO consultant since 2005. Last updated 8 November 2026.

How the SEO Spider works

The SEO Spider runs on Windows, macOS and Linux. You enter a starting URL, and it follows links to discover pages, images, scripts and stylesheets, just as a web crawler would. Results appear in tabs and filters, and everything can be exported to spreadsheets.

It can also crawl a list of URLs, such as an XML sitemap, which is useful for checking redirects after a migration.

Free and paid versions

The free version crawls up to 500 URLs and covers many core checks. A paid annual licence removes the URL limit (subject to your computer’s memory or storage) and unlocks features such as:

  • JavaScript rendering, to see pages as a browser would.
  • Custom extraction, to scrape specific data from pages using XPath, CSS selectors or regex.
  • Connections to Google Analytics, Google Search Console and PageSpeed Insights.
  • Saving crawls, comparing crawls over time and scheduled crawls.
  • Spelling and grammar checks and accessibility checks.

What it can find

  • 404 errors and other broken links, with the pages linking to them.
  • Redirect chains and loops.
  • Missing, duplicate or overlong page titles and meta descriptions.
  • Pages with noindex tags, canonical issues or blocked by robots.txt.
  • Click depth problems, where important pages sit too far from the homepage.
  • Orphan pages, when combined with sitemap, analytics or Search Console data.
  • Large images, missing alt text and slow-responding URLs.

Each issue can be exported with the affected URLs, which makes it easy to share fix lists with developers or content teams.

Tips for using it well

  1. Check the configuration before crawling: user agent, JavaScript rendering, crawl limits and whether to respect robots.txt.
  2. Lower the crawl speed on fragile or shared servers to avoid overloading them.
  3. Use database storage mode for large sites rather than memory mode.
  4. Connect Search Console data to spot indexable pages that get no impressions.
  5. Focus on issues that affect important pages first, rather than fixing every warning.
  6. Save each crawl so you can compare before and after changes.

Remember that a crawler only sees what it can reach by following links or from the lists you give it. Pages behind logins, forms or blocked scripts may be missed, so treat the results as a strong guide rather than a complete map of how Google sees your site.

Related products and alternatives

Screaming Frog also makes a Log File Analyser, which helps with log file analysis by showing how search engine bots actually crawl a site.

Other SEO crawlers include Sitebulb and the cloud-based audit tools built into larger SEO platforms. Many SEOs use Screaming Frog alongside these, as a fast, flexible desktop option for detailed investigation.

Key Takeaways

  • Screaming Frog SEO Spider is a desktop crawler used for technical SEO audits.
  • The free version crawls up to 500 URLs, and a paid licence removes the limit.
  • It finds broken links, redirects, duplicate titles, indexing issues and more.
  • Paid features include JavaScript rendering, custom extraction and API integrations.
  • Configure crawl settings carefully and prioritise issues on important pages.

Related Terms

Curious how this applies to your website? Request a free SEO check-up, or keep learning in the SEO Encyclopedia.