Free Robots.txt Tester & Checker
Test and validate any robots.txt file in real time. Verify whether Googlebot, Bingbot, and AI bots like GPTBot or Claude can crawl your URLs, check syntax errors, and test live changes in your browser.
Test and validate your robots.txt
Fetch a live robots.txt from your website or paste custom directives to test crawl access before deploying.
Trusted worldwide
Trusted by thousands of SEO pros
From solo consultants to global agencies, teams choose TrueRanker to unify SEO and AI-search visibility.
How to test and validate your robots.txt in three steps
Fetch any live robots.txt from a domain or paste your draft file directly to test crawl rules before deploying.
Enter your domain or paste rules
Provide your website URL to fetch the live robots.txt, or switch to paste mode to test uncommitted draft directives before pushing to production.
Test specific URL paths and user-agents
Choose a crawler like Googlebot, Bingbot, GPTBot, or ClaudeBot, enter a target path or URL, and instantly check if it is Allowed or Disallowed per RFC 9309.
Audit syntax issues and edit live
Inspect syntax errors, deprecated directives, and declared sitemaps. Edit the file directly in the online editor, re-test instantly, and download your clean robots.txt.
What the tester evaluates in your robots.txt
Six essential checks performed instantly against modern search engine standards and RFC 9309 specifications.
RFC 9309 syntax validation
Identifies missing colons, orphan rules declared before a User-agent, unrecognised directives, and invalid rule formatting across every line.
Longest-match rule precedence
Implements RFC 9309 Section 2.2.2: the longest matching pattern wins, and matching Allow directives override Disallow when pattern lengths are equal.
Wildcard and end-anchor parsing
Accurately tests * (matches any sequence of characters) and $ (end-of-URL path anchor) against query strings and parameter URLs.
Search vs AI bot matrix
Simultaneously audits access rules for traditional search engines (Googlebot, Bingbot) and generative AI scrapers (GPTBot, ClaudeBot, PerplexityBot).
XML Sitemap declarations
Detects and validates all Sitemap: URLs declared in the file so search crawlers can discover your canonical indexation map.
Site-wide block & size warnings
Warns if a global Disallow: / rule accidentally shuts crawlers out of your entire domain, or if the file exceeds the 512 KiB Google limit.
robots.txt controls crawl access, not indexation
A common misconception in technical SEO is assuming that Disallow in robots.txt removes a page from Google’s index. It does not. A disallow rule simply forbids search engine bots from downloading the page content.
If a disallowed page has incoming internal or external links, Google can still index its URL, title, and anchor text without crawling its body. To completely prevent indexation, the page must be accessible so Googlebot can read a noindex meta robots tag or X-Robots-Tag HTTP header.
Since September 2019, Google explicitly does not support noindex directives written inside robots.txt. This tester flags deprecated noindex and crawl-delay lines so your directives match modern standards.
- robots.txt manages crawl budget and server bandwidth
- Disallowing a URL does not guarantee exclusion from search results
- Use meta robots noindex on crawled pages to prevent indexation
- Sitemap declarations help search engines discover URLs cleanly
Common robots.txt pitfalls
- Using Disallow to hide pages from indexation
- Writing noindex directives in robots.txt (unsupported by Google)
- Accidentally blocking CSS, JS, or image resources necessary for rendering
- Blocking Googlebot while intending to block only AI scrapers
Beyond robots.txt: complete rank tracking and technical SEO
Validating your robots.txt file ensures crawlers can access your pages. But knowing how those accessible pages perform in search results and AI answer engines requires continuous monitoring.
TrueRanker tracks your daily keyword rankings across Google desktop and mobile, audits on-page technical issues, monitors backlinks, and measures your brand visibility inside AI engines like ChatGPT, Claude, and Gemini.
- Daily geolocated rank tracking down to city and postal code level
- Automated site audit for broken links, duplicate tags, and crawl errors
- AI Visibility tracking for brand mentions and source citations in LLMs
- Instant alerts when critical URLs drop or rankings change
What TrueRanker tracks
- Daily Google rankings with SERP feature detection
- Brand mentions and citations in ChatGPT, Claude, and Gemini
- Backlink gains, losses, and referring domain health
- On-page technical SEO errors across all your URLs
What SEO professionals and agencies using TrueRanker say
“With TrueRanker, I manage the SEO for my projects. It’s an essential tool for daily monitoring the impact of our On-Page changes and link-building strategies. ✌🏼”
“I don’t know how I lived without TrueRanker. It has become my daily SEO rank tracker for all my projects and Chrome extensions, something no other tool can do. I highly recommend it!”
“TrueRanker is a top tool that I use with clients and my own projects because what is not measured is not improved.”
“The best tool for tracking, most friendly support and opened for changes and improvments. You can generate something from here… and what I said is really true.”
“I started using TrueRanker in 2022 for keyword tracking, and today it’s one of my main SEO tools. It accurately handles rankings, competitor tracking, SEO audits, and AI visibility, replacing more expensive alternatives. Plus, their customer support is excellent. Highly recommended!”
“We benchmark keyword tracking tools regularly, and for over three years, TrueRanker has been our SEO agency’s top choice. Its intuitive interface, scalable plans, and responsive support offer everything we need to monitor client rankings accurately.”
“I’ve been using TrueRanker since August on my own blog. What I value most is that it tells me every day what has happened with what I’ve already published, and the GEO score of the audits showed me that no Google AI summary was citing me. In my own language, with support close by, and a trial without a card to check it before paying.”
Frequently asked questions about robots.txt testing
Everything you need to know about robots.txt directives, Googlebot crawling, AI bots, and troubleshooting crawl rules.
What is a robots.txt file and what does it do?
A robots.txt file is a plain text file placed at the root of your domain (e.g., example.com/robots.txt) following the Robots Exclusion Protocol (RFC 9309). It instructs automated web crawlers which URLs they may or may not request from your site, helping preserve server bandwidth and crawl budget.
How do I test if a URL is blocked by robots.txt?
Enter your domain in the tester, select the crawler (such as Googlebot or Bingbot), and input the URL path you want to verify (e.g., /blog/post-1 or /admin/). The tool tests your path against all applicable Allow and Disallow rules using RFC 9309 longest-match precedence and displays whether the URL is Allowed or Blocked along with the exact rule line.
How do I submit an updated robots.txt to Google?
Google automatically recrawls and refreshes your robots.txt file multiple times a day. If you make urgent updates and want Googlebot to fetch the new version immediately, open Google Search Console, navigate to Settings > robots.txt, and request a recrawl using the "Submit" or recrawl request feature.
What happens if an Allow and a Disallow rule match the same URL?
Under RFC 9309 specifications, the directive with the longest character pattern wins. For example, with "Disallow: /catalog" (9 chars) and "Allow: /catalog/item" (13 chars), the URL /catalog/item is allowed. If both rules have the exact same length (e.g., "Allow: /page" and "Disallow: /page"), the Allow directive takes precedence.
How do I block AI scrapers without blocking Google Search?
You can target specific AI crawler tokens in separate User-agent groups without affecting Googlebot. For example, declare "User-agent: GPTBot", "User-agent: ClaudeBot", or "User-agent: CCBot" with "Disallow: /". Googlebot will continue crawling your site under its own group or the wildcard User-agent: * group.
Can I use "Noindex" inside robots.txt to remove pages from Google?
No. Google officially ended support for the noindex directive in robots.txt in September 2019. If you include Noindex in robots.txt, Googlebot will simply ignore it. To remove or prevent pages from appearing in Google, use a <meta name="robots" content="noindex"> tag or an X-Robots-Tag HTTP header on a page that crawlers are allowed to fetch.
Does Googlebot obey the Crawl-delay directive?
No. Googlebot ignores Crawl-delay. If you need to manage Googlebot crawl rate, configure crawl rate settings directly in Google Search Console or use server-level HTTP rate limiting (such as 429 Too Many Requests status codes). Bingbot, however, does respect Crawl-delay in whole seconds.
Where must the robots.txt file be hosted?
The robots.txt file must reside at the exact top-level root of your website’s origin, accessible via HTTP/HTTPS at /robots.txt (e.g., https://example.com/robots.txt). Subdirectory locations such as example.com/site/robots.txt are not recognized by web crawlers.
Are robots.txt directives and URL paths case-sensitive?
Directive names (User-agent, Allow, Disallow) are case-insensitive. However, URL path patterns are case-sensitive. "Disallow: /Admin" will block /Admin/dashboard but will NOT match /admin/dashboard on standard web servers.
Ready for full visibility into your search rankings?
Track your keyword rankings daily, monitor backlinks, and audit technical SEO issues with TrueRanker. Start your 14-day free trial.