All checks
Crawlability · can they found?

robots.txt

A plain text file at yoursite.com/robots.txt that tells automated visitors — Google, Bing, ChatGPT — which parts of your site they may look at.

Why it matters

It is the first file almost every crawler requests. One over-broad line here can hide your entire site from search and from AI assistants, and nothing else on the page can compensate for it.

How to fix it, step by step

Quick fix — usually under ten minutes, no developer needed.

  1. 1Open yoursite.com/robots.txt in a browser.
  2. 2If you see "Disallow: /" under "User-agent: *", that single line blocks your whole site. Remove it.
  3. 3If the file does not exist, create one containing: User-agent: * then on the next line Allow: /
  4. 4Add a final line pointing at your sitemap: Sitemap: https://yoursite.com/sitemap.xml
  5. 5In WordPress, Yoast and Rank Math both let you edit this under Tools → File editor, with no FTP access needed.

How to know it worked

Reload yoursite.com/robots.txt and confirm no "Disallow: /" line remains. In Google Search Console, use the robots.txt report to confirm your homepage is allowed.

Others in Crawlability

Want to know which of these apply to your site?

The free scan runs all 33 checks against your pages and tells you which are failing, with the answer beside each one. No signup for the check itself.

Scan my site free