Latest—
robots.txt for a Small Business Website: A Copy-Paste Guide
A robots.txt file tells compliant search crawlers which URLs they may request. It does not make a page private or remove it from search results. For most small business sites the safe default is short: allow crawling, name your XML sitemap, and block nothing else. Use a page-level noindex tag for a page that must stay out of Google, and do not also block that page in robots.txt or Google will never see the tag. Never ship Disallow: / to a live site.
Read article →
robots.txt
technical seo
small business website
