robots.txt for a Small Business Website: A Copy-Paste Guide

Most small business sites do not need an elaborate robots.txt file. They need a short, boring one that doesn't accidentally lock search engines out of the pages that bring in customers.

Obvious, and still one of the quieter ways a small site falls out of Google. Usually it's a staging rule that shipped with the launch. Sometimes it's a snippet copied off a Shopify forum onto a WordPress brochure site, blocking paths that don't exist and missing the ones that do. Either way the first symptom is Search Console reporting "Indexed, though blocked by robots.txt" on a page you very much wanted crawled.

Copy the basic template below if it fits. If a line doesn't mean anything to you, that's a good sign you should leave it alone.

The short answer

robots.txt is a plain-text file at https://yourdomain.com/robots.txt. It tells search crawlers which URLs they may request. It is mainly for managing crawling, not for hiding pages or removing them from Google. Google can still show a blocked URL in search results if it finds the URL elsewhere.

For a normal brochure site, service business, portfolio, or local shop, this is usually enough:

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Replace www.example.com with your real canonical domain. On a hosted builder like Wix or Squarespace, use the platform's own settings rather than trying to upload a second file. There can only be one file at the root URL.

What it actually controls

When a crawler reaches your site, it normally checks /robots.txt before asking for other URLs. The rules tell a compliant crawler where it shouldn't go. That is useful for a cart, account pages, internal site-search results, or pages that produce endless filter combinations.

It is a request, not a lock. It won't password-protect anything, won't stop a person pasting the URL into a browser, and won't pull a page out of Google's results. Plenty of bots skip the file entirely, because scrapers have no reason to read it.

Google draws a hard line between crawling and indexing: a blocked URL can still be indexed if Google discovers it through links. The search result may then appear without a useful description. If a page must stay out of Google, Google needs to be able to crawl the page and see a noindex instruction, or the page needs real access control. Google's robots.txt introduction explains the difference in more detail.

A safe default for most small business sites

Use this if your public pages should be searchable and you have a sitemap:

# Let standard crawlers access public pages.
User-agent: *
Allow: /

# Replace this with your real sitemap URL.
Sitemap: https://www.example.com/sitemap.xml

The User-agent: * line means "all crawlers." Allow: / means they may crawl everything below the root. In practice, an empty robots.txt file also allows crawling, but writing the permission out makes the file easier for the next person to understand.

The sitemap line is optional, but useful. It points crawlers to the list of URLs you want indexed. Which URL you use depends on what built the site: /sitemap.xml on Shopify, Squarespace, and Wix, /wp-sitemap.xml on stock WordPress since 5.5, and /sitemap_index.xml if Yoast or Rank Math is running. Open yours in a browser before you paste it in. A sitemap helps discovery, though it doesn't guarantee that every URL will be indexed.

If your site has no sitemap yet, remove that line. Don't guess at the URL.

Copy-paste templates

A simple service business or portfolio

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Most sites I look at should be running exactly this and nothing else. Every public page stays crawlable, which is what you want.

A site with internal search results

Internal search pages can create thousands of thin URLs such as /search?q=blue+widgets. If those pages don't offer useful standalone search results, blocking their crawl path can make sense:

User-agent: *
Disallow: /search
Disallow: /*?s=

Sitemap: https://www.example.com/sitemap.xml

Note the missing trailing slash on the first rule. Disallow: /search/ would only match URLs where the next character is a slash, so it would sail straight past /search?q=blue+widgets, which is the exact URL shape you are trying to catch. The second rule covers WordPress, which puts its search term in a ?s= query parameter instead of a path. Check which shape your own site produces before you copy either line, and be aware that Disallow: /search also catches anything else starting with those seven characters, /search-tips included.

A store with a cart and customer account area

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Sitemap: https://www.example.com/sitemap.xml

Again, no trailing slashes: Shopify's own default file uses /cart, /checkout, and /account, and a rule written with a slash misses all three. WooCommerce calls its account area /my-account/, so adjust accordingly.

Before you paste any of this into a hosted platform, find out what that platform lets you do:

• Squarespace does not let you edit robots.txt at all. Every Squarespace site serves the same file. Nothing in this section applies to you.
• Shopify is not a settings screen. You override the file by adding templates/robots.txt.liquid in the theme code editor, which Shopify documents as an unsupported customization. This matters more than it sounds: a plain-text override replaces the entire default file, and Shopify's default ships roughly 35 Disallow rules covering /admin, /checkouts/, /policies/, and every sort_by collection URL. Paste a four-line template over it and you have silently deleted all of them.
• Wix has a real editor, under SEO & GEO in the dashboard, with a reset to default if you break something.
• WordPress has no editor in core. Yoast and Rank Math both provide one, or you can upload a physical robots.txt to the web root, which takes precedence over the virtual file WordPress generates.

Blocking a folder except for one file

Allow can make an exception inside a blocked folder:

User-agent: *
Disallow: /private/
Allow: /private/public-brochure.pdf

Sitemap: https://www.example.com/sitemap.xml

Two or three of these are readable. Ten and nobody will be able to tell you what the file does.

The directives you will actually use

There are more directives than this. You will not need them.

Rule matching is case-sensitive, and that is a property of the standard rather than of your server. Disallow: /Images/ will not block /images/logo.png even on a host that happily serves the page at both spellings, which is exactly the trap: the page looks blocked, the rule never fires, and nothing in Search Console tells you why. The path must start with / and is relative to the domain where the file lives. The standard also defines * for zero or more of any character and $ for the end of a URL, and every major crawler supports both, but they are an easy way to over-match a URL, so skip them until you have a specific pattern to solve. The Google robots.txt specification has the exact matching rules when you need them.

robots.txt versus noindex

The header works on HTML pages too. It is just the only option for files that have no <head> to put a meta tag in.

Google has to fetch a page before it can see its noindex tag. So don't block the same page in robots.txt while relying on noindex to remove it: remove the disallow rule first, make sure the page returns a normal response, add noindex, then give Google time to recrawl. See Google's noindex documentation for the supported formats, and the meta tags reference for where the tag goes.

Mistakes that can hurt a site

Blocking the whole site

User-agent: *
Disallow: /

That slash means every URL below your domain. Fine on staging. On a live site it is the single most expensive line you can ship: Google stops requesting your pages, and the ones already indexed decay out of the results while the rule sits there. Google does keep fetching robots.txt itself, so publishing a corrected file is enough to restart crawling. Just don't reach for it as a way to hide a launch temporarily. Password-protect the staging site instead.

Blocking CSS, JavaScript, or images needed to render pages

Guides written before Google started rendering pages often told people to block /css/ and /js/. Don't. Google renders your pages the way a browser does, and it needs those files to see the layout and the mobile version. Block an asset only when you know it has no role in rendering public pages.

This one bites modern JavaScript sites in a specific way. A Next.js site serves its stylesheets and scripts from /_next/, so a tidy-looking Disallow: /_next/ blocks the rendering of every page on the domain. If you must block the directory, carve out the assets:

Disallow: /_next/
Allow: /_next/static/

Using robots.txt for secret information

Never put a private folder name in robots.txt and assume it is hidden. The file is public. Anyone can open /robots.txt, and a crawler that ignores the rules can still request the URL. Protect customer documents, admin tools, invoices, and staging sites with authentication.

Copying rules without checking the real URL

Disallow: /blog also matches every path starting with /blog, including a page such as /blogging-tips/. That is how the standard defines matching, not a quirk of one crawler, so assume every conformant bot behaves this way. A trailing slash is usually a safer way to express a folder: Disallow: /blog/. The catch is that the slash is not always what you want, as the internal search template above shows. Test a few real URLs either way, including one you expect to allow.

Adding noindex to robots.txt

Google does not support noindex as a robots.txt directive. Put it in an HTML meta tag or an X-Robots-Tag response header instead.

Serving a server error on robots.txt

A 4xx on /robots.txt is survivable. A 5xx is not. If the file returns a server error, Google stops crawling the site for roughly the first 12 hours, then falls back to the last good copy it saw for up to 30 days. A plugin or firewall throwing a 503 at Googlebot can therefore keep an old, wrong file in force for a month after you fixed it.

Leaving a bad temporary rule in place

Google generally caches robots.txt for up to 24 hours, sometimes longer when it cannot refresh the file, so a correction is not always visible immediately. Publish the fix, make sure the file returns HTTP 200, then open the robots.txt report in Search Console and request a recrawl. That is faster than changing the rules again and waiting to see what happens.

Where to put the file

The file must live at the root of the domain:

https://www.example.com/robots.txt

These will not work as your site's robots file:

https://www.example.com/files/robots.txt
https://www.example.com/robots.txt.txt

The rules apply only to the same protocol, host, and port. A file on www.example.com does not govern example.com or shop.example.com. If you run separate subdomains, each one needs its own robots.txt file, or a redirect to the canonical one. Crawlers follow at least five redirects and apply the rules they end up with to the host they started from, so the usual apex-to-www redirect is fine.

Use a UTF-8 plain-text file named exactly robots.txt. It should return a 200 status code. Google treats most 4xx responses as though no robots.txt file exists, so a missing file is usually less damaging than a valid file that blocks the wrong URLs.

How to check your file before and after publishing

1. Open https://yourdomain.com/robots.txt in an incognito browser window. You should see plain text, not an HTML error page. A redirect or two to your canonical host is fine; a long chain is not.
2. Confirm the sitemap URL opens and contains your important public URLs.
3. Make a short list of URLs you expect to allow and URLs you expect to block.
4. In Search Console, open Settings and then the robots.txt report. Check that the fetch status is OK and the parse errors count is zero. This report replaced the old robots.txt Tester.
5. Run URL Inspection on one URL from each of your two lists and click Test live URL. The "Crawl allowed?" line answers against the file that is live right now, so you do not have to wait for a recrawl.
6. If you changed an existing file, keep a copy of the old version. Reverting a bad rule is much faster when you know what changed.

You are not building a crawler policy here. You are checking that a two-line change did not block something you needed crawled.

Generate a clean starting file

If you know which paths need blocking but don't want to assemble the syntax by hand, use the Brandize robots.txt generator. It gives you a clean file to review before you publish it. Generate it, open the live URL once it is deployed, and then forget about it until your URLs change.

While you are in the neighbourhood of your site's crawl settings, the meta tags reference covers the page-level tags that do the jobs robots.txt cannot, and how to make an OG image from your logo covers the one crawlers read but never index.