Looking for a newsgroup about steering web crawlers

  • Thread starter Thread starter Stan Hilliard
  • Start date Start date
S

Stan Hilliard

I looked at the log files of my Frontpage web. I am not expert in
reading log files but it is obvious that robots continuously search my
site. They go where I don't think they need to go. Like hit counter
files, obsolete folders, gif files, _vti folders, etc.

I need to learn what to put initially in a robots.txt file to provide
instruction to the robots. Is this the right place to discuss that or
is there another newsgroup for it?

Advice will be appreciated,
Stan Hilliard
 
This should get you started:
http://www.robotstxt.org/wc/robots.html
--
===
Tom Willett
Microsoft MVP - FrontPage
---
FrontPage Support:
http://www.frontpagemvps.com/
===
|I looked at the log files of my Frontpage web. I am not expert in
| reading log files but it is obvious that robots continuously search my
| site. They go where I don't think they need to go. Like hit counter
| files, obsolete folders, gif files, _vti folders, etc.
|
| I need to learn what to put initially in a robots.txt file to provide
| instruction to the robots. Is this the right place to discuss that or
| is there another newsgroup for it?
|
| Advice will be appreciated,
| Stan Hilliard
 
Back
Top