SEO Glossary

Robots.txt

A small text file at the root of a site that tells crawlers which parts they may visit. It controls crawling, not indexing — a page blocked here can still appear in results, just without its content read.

What robots.txt is

robots.txt is a plain text file that lives at one fixed address — yoursite.com/robots.txt — and sets the rules of engagement for crawlers:

User-agent: *
Disallow: /admin/
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

User-agent names which crawler the rules apply to (* means all), Disallow marks paths they may not fetch, and Sitemap points to your sitemap. Before crawling your site, Google checks this file.

Crawling is not indexing

The most misunderstood fact in technical SEO: robots.txt controls crawling — whether Google may fetch a page — not indexing, whether the URL may appear in results. A URL blocked by robots.txt can still show up in Google, learned from links elsewhere, displayed with no description because Google was never allowed to read it.

To keep a page out of results, use noindex and leave it crawlable. Use robots.txt for what it's for: keeping crawlers out of infinite calendar pages, faceted-filter mazes, cart flows, and other places where crawling is wasteful.

The failure modes

  • An overbroad rule. Disallow: / blocks the entire site — a single character of distance from harmless. Rules also apply by prefix, so Disallow: /blog blocks /blog-post-1 too, not just the /blog/ directory.
  • A robots.txt that errors. A missing file is fine (everything may be crawled). But when the file answers with a server error (5xx), Google can't tell what the rules are — its documentation describes treating the site cautiously, which in practice pauses crawling until the file recovers. A broken robots.txt quietly stalls the whole site's crawling.
  • Blocking assets. Disallowing your CSS or JavaScript prevents Google from rendering pages the way visitors see them, which can hurt how pages are evaluated.

Sanity-checking it

Read the file at /robots.txt and ask one question per important page: does any rule's prefix match this URL for User-agent: * or Googlebot? Search Console's robots.txt report shows the file Google last fetched and whether it parsed.

← Browse all terms