Open SEO Crawler

Website URL

Max pages: 1500

Workers: 5 concurrent

Per-host delay: 0.4s

Max depth: 10

User agent

Render JS (Playwright, slower) Solve bot challenges (headed browser fallback; slow, only on 403 challenge) Ignore robots.txt Ignore noindex (audit noindex pages as if indexable) Sitemap analysis (after crawl: missing / orphan / sitemap-only / etc.) Near-duplicate content (content body only; Shingle Jaccard)

Similarity ≥ %

Exclude CSS selectors (advanced)

Stops shared CTAs / testimonials / footers from inflating similarity. Canonicalised URLs are skipped automatically.

URL filters (robots.txt + custom rules)

Live robots.txt Add your own rules below. Syntax: * = any chars, $ = end-of-URL anchor. e.g. /*?* skip query strings · /page/* skip pagination · /*.pdf$ skip PDFs. Exclude patterns
One per line Include patterns
If set, URLs must match at least one

Summary

Errors

Warnings

Info

All Pages0

Issues

AI Crawlers Blocked0

AI Training Bots Blocked0

Search Engines Blocked0

Missing Meta Desc0

Meta Too Long0

Meta Too Short0

Missing Title0

Title Too Long0

Title Too Short0

Missing H10

Multiple H1s0

H1 Same as Title0

Missing Canonical0

Images Missing Alt0

All Images0

External Links0

No Schema Markup0

Schema by Page0

Thin Content0

Slow Response0

Bulk Reports

All Titles0

All Meta Descriptions0

All H1s0

All Canonicals0

Duplicate Titles0

Duplicate Meta Descriptions0

Duplicate H1s0

Duplicate Body Content0

Redirects & Status

Redirects0

Redirect Chains0

Soft 404 / URL Trap0

HTTP Errors (4xx/5xx)0

Response Codes0

Noindex Pages0

Canonicalised0

Deep Pages (4+ clicks)0

Hreflang0

Social & Security

Missing Open Graph0

Missing og:image0

Missing Twitter Card0

Missing Viewport0

Mixed Content0

URL Hygiene0

Sitemap analysis

Visualisations

Site Structure—

Crawl a site for technical SEO issues

Every internal page is audited — titles, meta, H1s, canonicals, broken links, thin & duplicate content, redirect chains, schema coverage, and AI-crawler blocks. Enter a URL and hit Start crawl.

Missing / duplicate meta Broken links Duplicate content Redirect chains Schema coverage AI-crawler blocks

Crawl a site for technical SEO issues

Crawl Budget

🔍 What is this?

🤖 Why add to robots.txt?

⚡ Why it helps crawl budget