What Robots.Txt Is & Why It Matters for SEO

4 July 2025 (Updated on 27 June 2026) 2 min read Technical SEO

What Is a Robots.txt File?

A robots.txt file is a set of instructions that tell search engines which pages to crawl and which pages to avoid, guiding crawler access but not necessarily keeping pages out of Google's index.

robots.txt syntax example

Robots.txt files, meta robots tags, and x-robots tags guide search engines in handling site content but differ in their level of control, whether they're located, and what they control.

Robots.txt: Site-wide instructions located in the root directory
Meta robots tags: Page-specific instructions in the <head>
X-robot tags: HTTP header instructions for non-HTML files

Further reading: Meta Robots Tag & X-Robots-Tag Explained

Why Is Robots.txt Important for SEO?

Robots.txt helps manage crawler activity and keeps bots focused on valuable content.

1. Optimize Crawl Budget

Prevent wasteful crawling of unimportant pages, especially on large sites.

2. Block Duplicate and Non-Public Pages

Useful for staging, login pages, or internal search.

3. Hide Resources

Exclude PDFs, media files, or sensitive directories.

How Does a Robots.txt File Work?

Bots look for /robots.txt before crawling.

User-agent: *
Disallow: /private

You can target specific bots or all with the * wildcard.

Instruction example: Allowing all except DuckDuckGo

Real-World Examples

YouTube

Blocks user content, feeds, and login pages.

YouTube robots.txt example

Nike

Blocks checkout and member inbox directories.

Nike robots.txt example

Understanding Robots.txt Syntax

Each block starts with a user-agent and defines rules like Allow or Disallow.

User-agent: *
Disallow: /admin
Allow: /admin/login.html

You can also add a Sitemap or Crawl-delay.

Sitemap directive example

Common Mistakes to Avoid

Placing robots.txt outside root
Using noindex in robots.txt (unsupported by Google)
Blocking important resources (JS/CSS)
Using absolute URLs instead of relative

Noindex tag example

How to Create and Test Robots.txt

1. Create the File

Use a text editor and save as "robots.txt".

2. Add Directives

Structure using groups per user-agent.

3. Upload

Upload to root directory of your site.

4. Test with Tools

Google Search Console, Semrush Site Audit, or open-source libraries.

Testing robots.txt format

Best Practices

One directive per line
Use wildcards where possible
Use "$" for end-matching
Add comments with #

Creative robots.txt comment examples

Conclusion

Your robots.txt file is a small but powerful tool. Done right, it improves crawl efficiency, preserves server resources, and keeps bots away from sensitive areas.

Test regularly, and fix any syntax or logic errors to keep your site in good standing.

External References

developers.google.com

Cluster Hub

This article is part of our crawl and indexation cluster. Read the full pillar guide: Optimising Crawl Budget: The Ultimate Technical SEO Guide.

Written by

Jimmy Harris

Technical SEO Specialist

Jimmy Harris is a technical SEO specialist focused on improving website performance, crawlability, and search visibility through practical, data-driven optimisation.

He works at the intersection of development and marketing, helping teams resolve complex technical issues such as site architecture, page speed, structured data, and indexing challenges. Jimmy specialises in translating SEO requirements into clear technical actions, ensuring websites are built in a way that search engines and users both understand.

With a strong background in performance optimisation and large-scale site audits, Jimmy takes a problem-solving approach to SEO, favouring measurable improvements over guesswork.

Technical SEO audits Site architecture and internal linking Core Web Vitals and performance optimisation Indexing and crawl budget management Structured data and schema implementation

View author profile