SEO

How to Create a Robots.txt File

Learn how to create a robots.txt file for your website. Step-by-step guide with copy-paste examples for common scenarios. Use our free generator to build yours in seconds.

2026-07-22 10 min read Beginner friendly

Quick answer

Create a plain-text file named robots.txt, add rules for the crawlers and paths you want to manage, then publish it at the root of your hostname. You can build and review the file with the Fileees robots.txt generator.

What Is a Robots.txt File?

A robots.txt file gives compliant web crawlers instructions about which URL paths they may crawl. It uses the Robots Exclusion Protocol and applies to one hostname and protocol. For example, rules at https://example.com/robots.txt do not automatically apply to a subdomain.

The file can reduce unnecessary crawling and point search engines to your sitemap. It does not remove a URL from search results, guarantee indexing, or prevent people from opening a public URL.

Does Your Website Need a Robots.txt File?

Not every website needs custom crawl rules. A small public site that allows all crawlers can work without a robots.txt file. Creating one is useful when you need to keep crawlers away from duplicate URL patterns, internal search pages, temporary sections, or large areas that add little search value.

You can also use robots.txt solely to advertise your XML sitemap. Build one with the sitemap generator, then add its absolute URL to a Sitemap: line.

Basic Robots.txt Syntax

User-agent: Selects the crawler that the following rules address. An asterisk targets all crawlers.
Disallow: Asks the selected crawler not to crawl matching paths. An empty value allows crawling.
Allow: Permits a more specific path inside a broader disallowed path when the crawler supports it.
Sitemap: Provides the absolute URL of an XML sitemap or sitemap index.

Paths are case-sensitive, comments begin with #, and rules should use URL paths rather than filesystem locations.

Create a Robots.txt File with Fileees

Open the Fileees robots.txt generator.
Enter your sitemap URL if you have one.
Add each path you want crawlers to allow or disallow.
Generate the file and review every rule before copying it.
Save the output as robots.txt and publish it at your site root.
Open the public URL in a browser and test important pages in your search engine tools.

Six Copy-Paste Robots.txt Examples

1. Allow all crawlers

User-agent: *
Disallow:

2. Block all crawling on a temporary host

User-agent: *
Disallow: /

Do not rely on this for private staging sites; use authentication as well.

3. Block an internal search area

User-agent: *
Disallow: /search/
Disallow: /*?query=

4. Block a folder but allow one public file

User-agent: *
Disallow: /documents/
Allow: /documents/public-guide.pdf

5. Add a sitemap

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

6. Give one crawler specific rules

User-agent: ExampleBot
Disallow: /archive/

User-agent: *
Disallow:

Where to Place Robots.txt

Upload the file to the root of the hostname it controls. For https://example.com, the correct address is https://example.com/robots.txt, not https://example.com/files/robots.txt. Serve it as plain text with an HTTP 200 response.

Each subdomain needs its own file. If both www.example.com and shop.example.com serve content, check the robots.txt URL on each hostname.

Common Robots.txt Mistakes

Mistake	Why it matters	Better approach
Using robots.txt as security	Blocked paths remain public and may still be discovered.	Use authentication or server access controls.
Blocking CSS or JavaScript	Search engines may not render the page correctly.	Allow assets needed for page rendering.
Publishing on the wrong path	Crawlers only look for the file at the hostname root.	Use `/robots.txt`.
Using a relative sitemap URL	The Sitemap directive should be an absolute URL.	Include protocol and hostname.
Assuming it removes indexed URLs	Disallowed URLs can remain in search results.	Use an appropriate index-control method while allowing recrawl.
Skipping tests after changes	A single slash can block an important section.	Review rules and test representative URLs.

Limitations and Privacy Notes

Robots.txt is not a security control. Compliant crawlers may follow it, but malicious bots can ignore it, and the file publicly reveals the paths you list. Never use it to protect credentials, customer data, private documents, or administration panels.

The Fileees generator creates the text in your browser. Review internal path names before sharing the output in a public ticket or repository.

Related SEO Checks

Robots.txt is one part of technical SEO. Use the meta tag generator to prepare page metadata, check how it may appear with the SERP preview tool, and follow the beginner SEO checks guide before publishing.

Frequently Asked Questions

Where should I put my robots.txt file?

Place it at the root of the hostname it controls, such as https://example.com/robots.txt. A file in a subdirectory does not control the whole site.

Can robots.txt keep a page private?

No. Robots.txt is crawl guidance, not access control. Protect private content with authentication or another server-side security control.

Should every website have a robots.txt file?

No. A small public site that allows all crawling may work without one. Add a file when you need explicit crawl rules or want to advertise a sitemap.

Can I list more than one sitemap?

Yes. Add a separate Sitemap line for each sitemap or reference a sitemap index.