Web-Site Mapping tool

  • Thread starter Thread starter Ashwin Kambli
  • Start date Start date
A

Ashwin Kambli

Hi,
I am looking for a tool that will map a given web-site and list out all the
files/folders under that website.
For example, if I want to map www.voicenow.com, the tool should list all the
sub-folders, and the files within each folder at www.voicenow.com.
More importantly, I should be able to use this tool programatically (even if
the tool generates an xml document of the site map, that will be useful)
from c#.
Please let me know of any such tool you might be aware of.
Thank you.
Ashwin Kambli
 
You cannot list all the files/folders of a website (unless the webserver
does a directory listing)... Think about security.

You can only crawl a site - with exposed links. For that, you have to write
your own crawler. It's not that difficult. You will have to parse an HTTP
page and strip out all links that belong to the same domain and add it to a
queue of "to-be-crawled" items. Make sure you don't go in a cyclic endless
loop.

-vJ
 
Back
Top