programmatically navigate websites

  • Thread starter Thread starter Tina
  • Start date Start date
T

Tina

How can I programmatically navigate a website just as a user would? click
on buttons, hyperlinks, etc. I know how to read nodes using XPath but I
don't know what classes to use to navigate.

I have to write a program to automatically read mail from an html mail
system.
Thanks,
T
 
it will depend on the website. you can use webclient to get the page.
fews sites render valid xml, so you will need an html parser to read the
page. some pages require javascript, or have javascript modifiy the dom.
this would be common with a mail system, but it may have a dumb mode
(maybe for mobile devices). for instance with aspx any thing but an
imagebutton or submit button require javascript to do postback. you will
need emulate this.

you should read the w3c http 1.1 protocol spec so you know how to get
and post data. some site (say an aspx site with viewstate), will require
you do a get and postback matching data.

you should get a tool like fiddler, so you can see the raw data passed
back and forth and what headers are required.



-- bruce (sqlwork.com)
 
Tina said:
How can I programmatically navigate a website just as a user would? click
on buttons, hyperlinks, etc. I know how to read nodes using XPath but I
don't know what classes to use to navigate.

I have to write a program to automatically read mail from an html mail
system.
Thanks,
T
Click to expand...
You have a few options:
* Automate IE - you can navigate to a page and use the DOM to fill out text
boxes, press buttons and read results
* Use MSXML2.XmlHttp.6.0 - you can make POST and GET calls and parse the
returned results using the 'HtmlFile' COM class to create an HTML document
* The same as above but using WebRequest and WebResponse from .NET
* Use a tool like Visual Studio - you can record macros that move you
through a site, then you refine them to use inputs from a database and make
decisions according to returned data
 
Back
Top