How to write a robot text file

Recipes

Camping

Scrapbook
albums

Guestbook/ newsletter

Archives

Sponsors

Physical Fitness

Events

Itinerary

Contribute

Job ads

Free rental ads

Privacy Policy

Calendars

Living Vicariously

Safety tips

Wut2do

Search

Associate

Send us a link

Link to Us

Table of contents

Contact us

See also "writing meta tags"

Robot text (robot.txt)

How to write a robot text file (robot.txt)

This is a short file created in notepad or equivalent, that tell the search engine spider bots whether they are allowed to and what to not to index. This file is copied and pasted into the root directory of your web.

Note that in order for a file or folder to be not be indexed then it must be listed in the disallow line. There is no such thing as an "allow" line. If it is not disallowed, then it is allowed. The asterisk * is a wildcard and indicates any search engine. The single / indicates the root directory and everything below it. Some folder should be protected if you have a sign up membership or have code you want protected.

This permits all search engine to index all pages on a site. and seems to be most popular.
User-agent: *
Disallow:

This excludes all robots from certain areas of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /stats/

Exclude a specific spider from parts of the server but not others
User-agent:grabber/1.5
Disallow: /cgi-bin/
Disallow:/stats/

To allow a single robot complete access and exclude all others
User-agent: Googlebot/1.0
Disallow:
User-agent: *
Disallow: /

This would prevent your entire web site from being indexed
User-agent: *
Disallow: /

You can and add the Robots meta tag to the <head> of your web page to instruct spiders what to index and what not to. Click the link above for a full explanation of the tag options.

<meta name="robots" content="index, follow">

You can use any one of these tags on specific pages according to your requirements for each page

Click here to make wut2c.com your homepage

Go to top Build an Itinerary

About wut2.com Advertise with us Link to Us Contact us Report Errors Job Opportunities Legal Copyright