GOOGLE NEWS PARSER

  • Thread starter Thread starter Krakatioison
  • Start date Start date
K

Krakatioison

I realized I am an idiot. Whole saturday wasted on this crazy regex crap, I
can't even look at it anymore...uff
Honestly, I am even willing to pay to someone who is able to solve this.
Please help, I really used all my knowledge on this and I cannot get it to
work.

Let me explain:

I need to parse, URL , URL TEXT and DESCRIPTION TEXT from GOOGLE NEWS.

Every headline in google news has the same structure and it looks like this:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q=http://www.canada.com/sports/st
ory.html%3Fid%3D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. <br>

go to lets say: http://news.google.com/news/en/us/sports.html and look up
the source if needed.


So, this is my parser (function) for grabbing links. And it works (thanks
God) and I am able to extract with this function all 20 major headlines URL
links:

Public Function ParseLinks(ByVal HTML As String) As ArrayList


Dim objRegEx As System.Text.RegularExpressions.Regex

Dim objMatch As System.Text.RegularExpressions.Match

Dim arrLinks As New System.Collections.ArrayList

objRegEx = New System.Text.RegularExpressions.Regex("(?:y
[hH][rR][eE][fF]\s*=)(?:[\s""']*)(?!#|[Mm]ailto|[lL]ocation.|[jJ]avascript|.
*css|.*this\.)(.*?)(?:[\s>""'])",
System.Text.RegularExpressions.RegexOptions.IgnoreCase Or
System.Text.RegularExpressions.RegexOptions.Compiled)

objMatch = objRegEx.Match(HTML)

While objMatch.Success

Dim strMatch As String

strMatch = objMatch.Groups(1).ToString

arrLinks.Add(strMatch)

objMatch = objMatch.NextMatch()

End While

Return arrLinks

End Function



Now as you probably guessed already, my problem is that I am simply not able
to write the same function for extraction of URL TEXT and DESCRIPTION TEXT.

please help

K.
 
Krakatioison,
What do you mean by url, url test and description text. show me which
portions you need.
Jared
 
Hi Jared,
so when you look at this google html:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q=http://www.canada.com/sports/st
ory.html%3Fid%3D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. <br>

This is what I need to get:

LINK:
http://news.google.com/url?ntc=05SA....html?id=5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782
LINKS TEXT: Everyone expected Ullrich to show up - instead it was teammate
Kloden
DESCRIPTION: BESANCON, France (AP) - Look for a German cyclist to be on the
Tour de France podium Sunday - just not the one most people expected.

That is what I need.
My parser gets the LINK.

But I need to get LINKS TEXT and DESCRIPTION.

God bless you men If you can extract it from the code.

K.
 
Guy's,
I'll pay us$50 to person who can write the regex command.
Looks like there is no other way.
I spent so much of my own time trying to write it without any success.
k.
 
Hi,

Why dont you use a news rss instead.

http://www.wired.com/news/rss

http://news.yahoo.com/rss

Google RSS

http://googlenews.74d.com/

http://www.google.com/newsalerts?q=news+rss&hl=en

Parse Web page

http://www.developerfusion.com/show/4415/


RSS Reader

http://www.philweber.com/articles/easy_rss_in_vbnet.htm



Ken
---------------------
I realized I am an idiot. Whole saturday wasted on this crazy regex crap, I
can't even look at it anymore...uff
Honestly, I am even willing to pay to someone who is able to solve this.
Please help, I really used all my knowledge on this and I cannot get it to
work.

Let me explain:

I need to parse, URL , URL TEXT and DESCRIPTION TEXT from GOOGLE NEWS.

Every headline in google news has the same structure and it looks like this:

<a class="y"
href="http://news.google.com/url?ntc=05SA0&q=http://www.canada.com/sports/st
ory.html%3Fid%3D5FBD7D23-AA7A-4E4C-AE1D-01CEBC350782"> Everyone expected
Ullrich to show up - instead it was teammate Kloden</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>
BESANCON, France (AP) - Look for a German cyclist to be on the Tour de
France podium Sunday - just not the one most people expected. <br>

go to lets say: http://news.google.com/news/en/us/sports.html and look up
the source if needed.


So, this is my parser (function) for grabbing links. And it works (thanks
God) and I am able to extract with this function all 20 major headlines URL
links:

Public Function ParseLinks(ByVal HTML As String) As ArrayList


Dim objRegEx As System.Text.RegularExpressions.Regex

Dim objMatch As System.Text.RegularExpressions.Match

Dim arrLinks As New System.Collections.ArrayList

objRegEx = New System.Text.RegularExpressions.Regex("(?:y
[hH][rR][eE][fF]\s*=)(?:[\s""']*)(?!#|[Mm]ailto|[lL]ocation.|[jJ]avascript|.
*css|.*this\.)(.*?)(?:[\s>""'])",
System.Text.RegularExpressions.RegexOptions.IgnoreCase Or
System.Text.RegularExpressions.RegexOptions.Compiled)

objMatch = objRegEx.Match(HTML)

While objMatch.Success

Dim strMatch As String

strMatch = objMatch.Groups(1).ToString

arrLinks.Add(strMatch)

objMatch = objMatch.NextMatch()

End While

Return arrLinks

End Function



Now as you probably guessed already, my problem is that I am simply not able
to write the same function for extraction of URL TEXT and DESCRIPTION TEXT.

please help

K.
 
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html


This is the html code:

<a class="y"
href=http://www.LINK.com>LINK TEXT</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>LI
NK DESCRIPTION <br>

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.
 
Hi,

http://www.developerfusion.com/show/4415/

Ken
---------------------
Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html


This is the html code:

<a class="y"
href=http://www.LINK.com>LINK TEXT</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>LI
NK DESCRIPTION <br>

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.
 
Ken,
I already tried it.
Their code is not build to extract from google.... it doesn't work.
k.
 
Hi,

Here is a start.

Dim wc As New System.Net.WebClient

Dim sr As New System.IO.StreamReader(wc.OpenRead("http://news.google.com/"))

Dim strHtml As String



Dim regLink As New
System.Text.RegularExpressions.Regex("\""(?<url>[^\""]*)\""")

Dim regTitle As New System.Text.RegularExpressions.Regex(">(.*?)\<")

Dim regHref As New System.Text.RegularExpressions.Regex("\<a
href=""(.*?)""\>(.*?)\<\/a\>")

Dim m As System.Text.RegularExpressions.Match

strHtml = sr.ReadToEnd

Try

For Each m In regHref.Matches(strHtml)

Dim mLink As System.Text.RegularExpressions.Match

For Each mLink In regLink.Matches(m.ToString())

Trace.WriteLine(String.Format("Link {0}", mLink.ToString))

Next

For Each mLink In regTitle.Matches(m.ToString())

Dim strTitle As String = mLink.ToString

strTitle = strTitle.Replace(">", "")

strTitle = strTitle.Replace("<", "")

Trace.WriteLine(String.Format("Title {0}", strTitle))

Next

Next

Catch

End Try

sr.Close()

wc.Dispose()



Ken

----------------------------

Ken thanks for your answer and your time.
It's very appreciated. Looks like no one else answered to this with some
solution.
But as I said, I don't want to get this in form of rss or anything like
that...
All I need is one line of code for regex to parse LINK, LINK TEXT,
DESCRIPTION out of this html


This is the html code:

<a class="y"
href=http://www.LINK.com>LINK TEXT</a><br><font size="-1"
style="font-family: arial,sans-serif"><b>
<font color="#6f6f6f" style="font-family:
arial,sans-serif">Canada.com&nbsp;-</font>15&nbsp;minutes&nbsp;ago</b><br>LI
NK DESCRIPTION <br>

In this case:
Link is: http://www.LINK.com
Link Text: LINK TEXT
Description: LINK DESCRIPTION

Does someone find a time to aswer me with one single line of regex code
which would do this?

Please.

K.
 
Krakatioison

All I need is one line of code for regex to parse LINK, >LINK TEXT,
Click to expand...
DESCRIPTION out of this html

</snip>

I think you are asking too much of Regex. Besides, even if you manage to
find the correct command, think of the next person who has to maintain
the code, including yourself. The following works, and is easier to
read. It will not be as fast, but you are only dealing with 20 links.

Public Function ParseGoogle(ByVal GoogleStr As String, ByRef URL As
String, ByRef URLText As String, ByRef Description As String) As Boolean
Dim HTTP_pos As Integer
Dim UrlEnd_pos As Integer
Dim UrlTextEnd_pos As Integer
Dim DscrStart_pos As Integer
Dim Result As Boolean

Result = True

HTTP_pos = InStr(1, GoogleStr, "http", CompareMethod.Text)
UrlEnd_pos = InStr(1, GoogleStr, ">", CompareMethod.Text)
UrlTextEnd_pos = InStr(1, GoogleStr, "</a>", CompareMethod.Text)
DscrStart_pos = InStr(1, GoogleStr, "<br>", CompareMethod.Text)
DscrStart_pos = InStr(DscrStart_pos + 1, GoogleStr, "<br>",
CompareMethod.Text)

URL = Mid$(GoogleStr, HTTP_pos, UrlEnd_pos - HTTP_pos - 1)
URLText = Mid$(GoogleStr, UrlEnd_pos + 1, UrlTextEnd_pos -
UrlEnd_pos - 1)
Description = Mid$(GoogleStr, DscrStart_pos + 4, Len(GoogleStr) -
DscrStart_pos - 8)

ParseGoogle = Result

End Function

Hope this helps.

James
 
Back
Top