Developing a C# Webcrawler

  • Thread starter Thread starter Logician
  • Start date Start date
L

Logician

I am looking for some help and I wondered if any C# gurus had some
suggestions.

I have tried contacting some Indian companies via email and also I
have placed projects at rentacoder.com and odesk.com. I lost 800 USD
at rentacoder after a man started the work and then just gave up
without anything completed.

I am using the code at Heaton Research for a basis - see
http://www.heatonresearch.com/download/. It seems developers find the
work too hard.
 
My suggestion is to at least explain what the problem is, not what problems
you've experienced along the way :-)
 
I lost 800 USD
atrentacoderafter a man started the work and then just gave up
without anything completed.
Click to expand...

I can't trust you because RentACoder has "Escrow deposit" feature.
Buyer release Escrow deposit when all work complete.
You can lose time, but you can't lost money.

Please, show me link to your profile at RentACoder.
 
My suggestion is to at least explain what the problem is, not what problems
you've experienced along the way :-)
Click to expand...

*OFF-TOPIC*

Just wanted to let you know this has become my quote of the day. I'm
going to print this and tape it up. Applicable to this situation and
so many more..:)
 
I can't trust you because RentACoder has "Escrow deposit" feature.
Buyer release Escrow deposit when all work complete.
You can lose time, but you can't lost money.

Please, show me link to your profile at RentACoder.
Click to expand...

Escrow works on a task basis and I split the larger task into smaller
ones, trusting the developer. Small tasks were paid as some work was
done. The larger task was never completed, and half-done
incomprehensive code is useless. Several other developers looked at
the half done code and said it was impossible to complete.
 
My suggestion is to at least explain what the problem is, not what problems
you've experienced along the way :-)
Click to expand...

The problem is that good people are in high demand and they earn a lot
of money. Getting them is hard. The work is generally too hard for
most developers. So the issue is one of finding someone who is
talented and willing to work for less money. Indian development is the
normal route under these conditions but I am finding that people just
find the work too hard to do.

This problem is fairly standard and that is why people take in venture
capital.
 
Escrow works on a task basis and I split the larger task into smaller
ones, trusting the developer. Small tasks were paid as some work was
done. The larger task was never completed, and half-done incomprehensive
code is useless. Several other developers looked at the half done code
and said it was impossible to complete.
Click to expand...

It is a rare programmer that can take a half done job and turn it into a
complete system. It is a knack being able to understand the intent of
code already written and then add what is needed. If the job is
halfbaked then you are better off throwing the whole thing.

If you spec a process that does one part completely and is testable then
get that done you are at least in the right direction.

Ken
 
Logician said:
The problem is that good people are in high demand and they earn a lot
of money. Getting them is hard. The work is generally too hard for
most developers. So the issue is one of finding someone who is
talented and willing to work for less money. Indian development is the
normal route under these conditions but I am finding that people just
find the work too hard to do.

This problem is fairly standard and that is why people take in venture
capital.
Click to expand...

If you want the best people you need to pay what they want !

Why should they work for less than their market value ?

Arne
 
Obviously a very outdated post...

Writing a basic crawler is easy. Writing one that is efficient and handles the large number of edge case conditions present is not, mainly due to the time involved in crawling and subsequent debugging. Your other developer probably gave up right around the time of discovering that simply using a regular expression to filter Files/Images/HyperLinks wasn't enough.

I offer two complete crawlers, with full source.

http://arachnode.net

Thanks,
Mike
 
Back
Top