Robots.txt

  • Thread starter Thread starter Mike Reilly
  • Start date Start date
M

Mike Reilly

I have a site that has been published in FP2002 and hosted on an Apache
server. When looking over the site analysis reports I noticed that in the
"Failures" section that /robots.txt is by far the most common failure.

I would like someone to look at the code on my site to see if my "Robots"
meta tag is done correctly.

The site is www.outsidebanks.com .

I would appreciate the help if something is wrong here. Google has stopped
crawling the site for some reason. Some other spiders are still visiting
regularly.

Thanks
 
It's not the meta tag, the search bots are looking
for a file named robots.txt which is used to set
permissions for the search engines / web crawlers.

Create one using notepad, that contains the following:
User-agent: *
and save it in your root directoryas robots.txtif you want it to exclude
your images folder then use this.User-agent: *
Disallow: /images/
Also, do a Google search for robots.txtfor more info
-- using 2k PRO but....95isaliveThis site is best
viewed...................................with a computer"Mike Reilly"
 
Reformatted.!!

It's not the meta tag, the search bots are looking
for a file named robots.txt which is used to set
permissions for the search engines / web crawlers.

Create one using notepad, that contains the following:

User-agent: *

and save it in your root directory as robots.txt
if you want it to exclude
your images folder then use this.

User-agent: *
Disallow: /images/

Also, do a Google search for robots.txt for more info



I have a site that has been published in FP2002 and hosted on an Apache
server. When looking over the site analysis reports I noticed that in the
"Failures" section that /robots.txt is by far the most common failure.

I would like someone to look at the code on my site to see if my "Robots"
meta tag is done correctly.

The site is www.outsidebanks.com .

I would appreciate the help if something is wrong here. Google has stopped
crawling the site for some reason. Some other spiders are still visiting
regularly.

Thanks
 
Back
Top