Web Scraping on Secured Sites that require UserName and Password.

  • Thread starter Thread starter John West
  • Start date Start date
J

John West

I have a Web Site that I want to monitor and gather
information from which requires a secured user id and
password of which I have but don't know how to gather get
by the secured log in to get to the pages I need. I have
gathered information from sites that don't require secured
login by using the following code:

Function readHtmlPage(ByVal url As String) As String
Dim objResponse As WebResponse
Dim objRequest As WebRequest
Dim result As String
objRequest = System.Net.HttpWebRequest.Create(url)
objResponse = objRequest.GetResponse()
Dim sr As New StreamReader
(objResponse.GetResponseStream())
result = sr.ReadToEnd()
sr.Close()
Return result
End Function

But don't know how to get around the sites that require
secured log-in such as https protocol.

Any help or direction would be appreciated.
 
Have you tried the Internet Transfer control? I use that ocassionally to get
FTP files. It might be a VB6/Legacy thing. It does work in .Net, though.
Fortunately, if you own VB.Net, I think you're licensed to use VB6 as well.
You'd just need to find a copy to install.
 
* "Microsoft said:
Have you tried the Internet Transfer control? I use that ocassionally to get
FTP files. It might be a VB6/Legacy thing. It does work in .Net, though.
Fortunately, if you own VB.Net, I think you're licensed to use VB6 as well.
You'd just need to find a copy to install.
Click to expand...

For FTP:

<http://www.freevbcode.com/ShowCode.Asp?ID=4655>
<http://www.allapi.net/downloads/NetFTP.zip>
<http://www.visualbuilder.com/article/viewarticle.asp?id=1234>
<http://www.abderaware.com/> (FTP component)
<http://www.csharphelp.com/archives/archive9.html>
<http://www.codeproject.com/csharp/FTPDriver1.asp>
....
 
Hi Herfried,

Can we do the Pentagon also with your help.
Nobody was answering this thread, but there is always one
.........................
:-((

Cor
 
Back
Top