I'm Missing robot.txt?

  • Thread starter Thread starter Guest
  • Start date Start date
G

Guest

I am using google sitemaps to try to help my site out. It says I have no
robot.txt file on my site, I thought this was always on it, I didn't know I
had to do that. How do I do this? Does it have to be on every pag or just the
index page somewhere?
 
Robots text is a file you create in Notepad and save it as robots.txt

All it needs to contain is:

User-agent: *
Disallow:

Then save it to the root of your web and publish.
(where your home page lives)


--
Steve Easton
Microsoft MVP FrontPage
95isalive
This site is best viewed..................
...............................with a computer
 
See this tutorial:
http://www.searchengineworld.com/robots/robots_tutorial.htm
--
===
Tom [Pepper] Willett
Microsoft MVP - FrontPage
---
FrontPage Support:
http://www.frontpagemvps.com/
===
|I am using google sitemaps to try to help my site out. It says I have no
| robot.txt file on my site, I thought this was always on it, I didn't know
I
| had to do that. How do I do this? Does it have to be on every pag or just
the
| index page somewhere?
| --
| kim
 
It is a simple file that you place in the root of your directory. Name it
robot.txt
in the file you should have text like:
User-agent: *
Disallow:

This will allow the search engine to search your entire site or you can add
this:
User-agent: *
Disallow: /cgi-bin/
Disallow: /ftp_tmp/

Which will allow all of your site to be searched except for cgi-bin or ftp_tmp

Hope this helps. Oh check out this site for more info:
http://www.robotstxt.org/
Peace, Tom
 
FYI
needs to be robots.txt not robot.txt


--

_____________________________________________
SBR @ ENJOY (-: [ Microsoft MVP - FrontPage ]
"Warning - Using the F1 Key will not break anything!" (-;
To find the best Newsgroup for FrontPage support see:
http://www.frontpagemvps.com/FrontPageNewsGroups/tabid/53/Default.aspx
_____________________________________________


| It is a simple file that you place in the root of your directory. Name it
| robot.txt
| in the file you should have text like:
| User-agent: *
| Disallow:
|
| This will allow the search engine to search your entire site or you can add
| this:
| User-agent: *
| Disallow: /cgi-bin/
| Disallow: /ftp_tmp/
|
| Which will allow all of your site to be searched except for cgi-bin or ftp_tmp
|
| Hope this helps. Oh check out this site for more info:
| http://www.robotstxt.org/
| Peace, Tom
|
|
| "Kim" wrote:
|
| > I am using google sitemaps to try to help my site out. It says I have no
| > robot.txt file on my site, I thought this was always on it, I didn't know I
| > had to do that. How do I do this? Does it have to be on every pag or just the
| > index page somewhere?
| > --
| > kim
 
I went to the tutorials, thank you all for your great advice. Does this mean
I just make a seperate folder named robot.txt or can I just through it in my
front page code near meta data? Yes, I have no clue what I am doing :-)

webmonger said:
You're right, I missed that. Thank you!
Peace,
Tom
Stefan B Rusynko said:
FYI
needs to be robots.txt not robot.txt


--

_____________________________________________
SBR @ ENJOY (-: [ Microsoft MVP - FrontPage ]
"Warning - Using the F1 Key will not break anything!" (-;
To find the best Newsgroup for FrontPage support see:
http://www.frontpagemvps.com/FrontPageNewsGroups/tabid/53/Default.aspx
_____________________________________________


| It is a simple file that you place in the root of your directory. Name it
| robot.txt
| in the file you should have text like:
| User-agent: *
| Disallow:
|
| This will allow the search engine to search your entire site or you can add
| this:
| User-agent: *
| Disallow: /cgi-bin/
| Disallow: /ftp_tmp/
|
| Which will allow all of your site to be searched except for cgi-bin or ftp_tmp
|
| Hope this helps. Oh check out this site for more info:
| http://www.robotstxt.org/
| Peace, Tom
|
|
| "Kim" wrote:
|
| > I am using google sitemaps to try to help my site out. It says I have no
| > robot.txt file on my site, I thought this was always on it, I didn't know I
| > had to do that. How do I do this? Does it have to be on every pag or just the
| > index page somewhere?
| > --
| > kim
Click to expand...
Click to expand...
 
You make the robots.txt file in notepad, and import it into the root of your
root web. No folders. No other files.

Remember, it's NOT robot.txt
--
Tom [Pepper] Willett
Microsoft MVP - FrontPage
FrontPage Support: http://www.frontpagemvps.com/
----------
Kim said:
I went to the tutorials, thank you all for your great advice. Does this
mean
I just make a seperate folder named robot.txt or can I just through it in
my
front page code near meta data? Yes, I have no clue what I am doing :-)

webmonger said:
You're right, I missed that. Thank you!
Peace,
Tom
Stefan B Rusynko said:
FYI
needs to be robots.txt not robot.txt


--

_____________________________________________
SBR @ ENJOY (-: [ Microsoft MVP - FrontPage ]
"Warning - Using the F1 Key will not break anything!" (-;
To find the best Newsgroup for FrontPage support see:

http://www.frontpagemvps.com/FrontPageNewsGroups/tabid/53/Default.aspx
_____________________________________________


| It is a simple file that you place in the root of your directory.
Name it
| robot.txt
| in the file you should have text like:
| User-agent: *
| Disallow:
|
| This will allow the search engine to search your entire site or you
can add
| this:
| User-agent: *
| Disallow: /cgi-bin/
| Disallow: /ftp_tmp/
|
| Which will allow all of your site to be searched except for cgi-bin
or ftp_tmp
|
| Hope this helps. Oh check out this site for more info:
| http://www.robotstxt.org/
| Peace, Tom
|
|
| "Kim" wrote:
|
| > I am using google sitemaps to try to help my site out. It says I
have no
| > robot.txt file on my site, I thought this was always on it, I
didn't know I
| > had to do that. How do I do this? Does it have to be on every pag
or just the
| > index page somewhere?
| > --
| > kim
Click to expand...
Click to expand...
Click to expand...
 
Back
Top