Sample AI Crawler Check

robots.txt AI crawler access for NovaDesk — illustrative demo, not a live workspace

www.novadesk.io

https://www.novadesk.io/robots.txt

robots.txt analysis for AI crawlers

Checked 2026/5/28 14:30

100Score

2 pass0 warn0 fail

Validation checklist

Parses live robots.txt — does not simulate per-URL fetches for each bot.

pass

robots.txt reachable

robots.txt returned HTTP 200.

pass

AI crawler access

No blanket Disallow detected for tracked AI crawlers.

AI crawler access

Per-bot status from robots.txt rules (blanket Disallow: /).

GPTBot

OpenAI

pass

ChatGPT-User

OpenAI

pass

OAI-SearchBot

OpenAI

pass

ClaudeBot

Anthropic

pass

PerplexityBot

Perplexity

pass

Google-Extended

Google

pass

CCBot

Common Crawl

pass

robots.txt preview

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://www.novadesk.io/sitemap.xml

Run a full GEO site audit

Sign in to score your brand homepage across six pillars — technical crawlability, schema, content, citations, and more — no credit card.

Get started free

Optimized for ChatGPT

More free AI content tools

Other standalone checkers — paste a URL, preview results in seconds, no account required.

AI Search Query Generator

Paste your brand website — get a search intent and five commercial-intent questions for AI search tracking.

Generate queries

AI Content Checker

Score any published article for topic clarity, structure, and quotable blocks — before you ship or rewrite.

Check content

AI Text Checker

Check AI text for GEO in 30 seconds — flag template phrasing that hurts readability, not plagiarism detection.

Check text

AI Text Humanizer

Humanize AI text for GEO in 30 seconds — Light, Balanced, or Thorough, then score structure in AI Content Checker.

Humanize text

GEO Content Writer

Try AI-optimized article writing free — brand, topic, and category in; short draft plus citation score out. No signup wall on results.

Write draft

LLMs.txt Checker

Verify your site has /llms.txt and validate structure for AI discovery — domain check or paste content.

Check llms.txt

LLMs.txt Generator

Generate a draft llms.txt from your public sitemap or homepage links — copy, deploy, and validate structure before going live.

Generate llms.txt

XML Sitemap Checker

Parse your XML sitemap or sitemap index — list URLs with metadata, surface issues, and check HTTP status on a sample.

Check sitemap

E-E-A-T Checker

Check author signals and About/Contact trust links on any URL — key cues AI systems use when citing sources.

Check E-E-A-T

AI Citation Checker

Scan any URL for outbound links, quotable blocks, and authority signals that help AI systems cite your page.

Check citations

Structured Data Checker

Validate JSON-LD, title, meta description, and FAQPage schema on any URL.

Check schema

View all free AI content tools