Download program

  • Thread starter Thread starter Bonj
  • Start date Start date
B

Bonj

Hello
I am often getting annoyed when the internet connection gets disconnected
and something is in the middle of a download, hence I am trying to find a
way of building a program that will access the URL of a download and then
transfer its bytes sequentially, but somehow remember how many bytes have
been transferred, in order that it can pick up where it left off. It
wouldn't matter if it was slightly slower to download but must be able to
download from any URL, not just programs served by a particular server
application. I'm guessing I will have to use some form of sockets based
programming so am unsure about whether it would be better to write this in
VB or C++, or .NET. So I hope you don't mind me cross posting as I would
appreciate suggestions as to how this could be done in any language.

Thanks for any suggestions

Cheers
 
Bonj said:
Hello
I am often getting annoyed when the internet connection gets disconnected
and something is in the middle of a download, hence I am trying to find a
way of building a program that will access the URL of a download and then
transfer its bytes sequentially, but somehow remember how many bytes have
been transferred, in order that it can pick up where it left off.
Click to expand...

No need! There are special programs for this called "download
managers." Here's an example of one - http://www.getright.com/

Further, as far as I know, the venerable Mozilla Firebird browser (
http://www.mozilla.org/products/firefox/ ) even has this functionality
in-built!
It
wouldn't matter if it was slightly slower to download
Click to expand...

I see no reason why this would be the case.
but must be able to
download from any URL, not just programs served by a particular server
application.
Click to expand...

It would have to adhere to HTTP and/or FTP. You can read up on these
protocols' RFCs.
I'm guessing I will have to use some form of sockets based
programming
Indeed.

so am unsure about whether it would be better to write this in
VB or C++, or .NET.
Click to expand...

It's all about your own preference here. Either will do.
So I hope you don't mind me cross posting as I would
appreciate suggestions as to how this could be done in any language.
Click to expand...

This can be done using "sockets."
Thanks for any suggestions

Cheers
Click to expand...

No problemo!
 
Well... You could use the InternetReadFile wininet API function to download
a file in chunks.

You'll also want these wininet functions...
- InternetOpen
- InternetOpenUrl
- InternetConnect
- HttpOpenRequest
- InternetCloseHandle

I've used these in VB5 for "auto online" updates. The updates aren't that
large (< 2 meg), so I just buffer in memory until the download's done, then
write it to disk. For what you're after, you'd probably want to write the
chunks to disk instead of buffering in memory. But then the "fun" would be
knowing whether you're continuing a previously interrupted download. You'd
probably need to create some sort of indexing scheme, a file (or in
registry) proprietary to your app that lists the URLs of downloads. Add the
URL to the list when the download starts, if the download completes remove
it from the list. You'd probably want to include info like URL, local
filename, filesize for each list item. Whenever a download starts, check if
the URL matches one in the list. If it does, then do some verification
(maybe comparing the first 1k chunk of the local file with that of the
remote one at the URL) to help assure its the same file.

Just some thoughts off the top of my head...
 
That's excellent thanks


BeastFish said:
Well... You could use the InternetReadFile wininet API function to download
a file in chunks.

You'll also want these wininet functions...
- InternetOpen
- InternetOpenUrl
- InternetConnect
- HttpOpenRequest
- InternetCloseHandle

I've used these in VB5 for "auto online" updates. The updates aren't that
large (< 2 meg), so I just buffer in memory until the download's done, then
write it to disk. For what you're after, you'd probably want to write the
chunks to disk instead of buffering in memory. But then the "fun" would be
knowing whether you're continuing a previously interrupted download. You'd
probably need to create some sort of indexing scheme, a file (or in
registry) proprietary to your app that lists the URLs of downloads. Add the
URL to the list when the download starts, if the download completes remove
it from the list. You'd probably want to include info like URL, local
filename, filesize for each list item. Whenever a download starts, check if
the URL matches one in the list. If it does, then do some verification
(maybe comparing the first 1k chunk of the local file with that of the
remote one at the URL) to help assure its the same file.

Just some thoughts off the top of my head...
Click to expand...
 
www.mozilla.com is the place to find FireFox and it is public
domain code as well, so you can download the source code
there. I downloaded it myself, and it's kind of hidden, but:

http://ftp24moz.newaol.com/pub/mozilla.org/firefox/

It's all c I believe. I haven't really dug into it yet. The browser
is really nice. Lots of folks giving rave reviews of it. Good
stuff!

--
Jim Carlock
http://www.microcosmotalk.com/
Post replies to the newsgroup.


C# Learner said:
[...]
Further, as far as I know, the venerable Mozilla Firebird browser (
http://www.mozilla.org/products/firefox/ ) even has this functionality
in-built!
[...]
Click to expand...

Mozilla Firefox*
 
Jim said:
www.mozilla.com is the place to find FireFox and it is public
domain code as well, so you can download the source code
there. I downloaded it myself, and it's kind of hidden, but:

http://ftp24moz.newaol.com/pub/mozilla.org/firefox/

It's all c I believe. I haven't really dug into it yet. The browser
is really nice. Lots of folks giving rave reviews of it. Good
stuff!
Click to expand...
Haven't used anything else in about a year.
(except when I'm forced to hit a site designed for MicroShaft browsers
only)

D>
 
Hello
I am often getting annoyed when the internet connection gets disconnected
and something is in the middle of a download, hence I am trying to find a
way of building a program that will access the URL of a download and then
transfer its bytes sequentially, but somehow remember how many bytes have
been transferred, in order that it can pick up where it left off. It
wouldn't matter if it was slightly slower to download but must be able to
download from any URL, not just programs served by a particular server
application. I'm guessing I will have to use some form of sockets based
programming so am unsure about whether it would be better to write this in
VB or C++, or .NET. So I hope you don't mind me cross posting as I would
appreciate suggestions as to how this could be done in any language.
Click to expand...

My understanding is that starting a download from anywhere other than
the start of the file is something that depends entirely on the
Server.

A Google on: FTP resume
should give you an idea of the problems
 
It didn't solve the problem I set out to solve but it seems a nicer browser
anyway, so I've installed it and am using it. It doesn't *seem* to be able
to resume a download from exactly where it left off like I intended, but it
does track the progress of all downloads in a central window and is still
able to resume, albeit possibly it starts again from the beginning of the
file.


Jim Carlock said:
www.mozilla.com is the place to find FireFox and it is public
domain code as well, so you can download the source code
there. I downloaded it myself, and it's kind of hidden, but:

http://ftp24moz.newaol.com/pub/mozilla.org/firefox/

It's all c I believe. I haven't really dug into it yet. The browser
is really nice. Lots of folks giving rave reviews of it. Good
stuff!

--
Jim Carlock
http://www.microcosmotalk.com/
Post replies to the newsgroup.


C# Learner said:
[...]
Further, as far as I know, the venerable Mozilla Firebird browser (
http://www.mozilla.org/products/firefox/ ) even has this functionality
in-built!
[...]
Click to expand...

Mozilla Firefox*
Click to expand...
 
Do you know how I would be able to navigate to the middle of a file, if I
knew the byte number I wanted to go to?
 
Bonj said:
It didn't solve the problem I set out to solve but it seems a nicer browser
anyway, so I've installed it and am using it. It doesn't *seem* to be able
to resume a download from exactly where it left off like I intended, but it
does track the progress of all downloads in a central window and is still
able to resume, albeit possibly it starts again from the beginning of the
file.
Click to expand...

I'd have to look further into this, but I'm sure I've seen Firefox
/resume/ downloads a couple of times, where the ISP connection has gone
down for some reason.

Don't forget, though, that the relevant server has to support the
ability to resume for this to work at all. Many do, but not all.
 
Most of what i got was 'how to write a successful resume', but I get the
picture - you're telling me it's completely impossible, without server
cooperation, right?
I can see why IE might not have implemented it - laziness, but if it was
possible Mozilla would have already done it wouldn't they? Damn!
 
BITS is not widely available on servers currently.

HTTP resume (or content ranges) is an HTTP 1.1 feature that allows you to
specify content ranges as part
of your headers so that you can get certain parts of the target resource.
Generally, you
start by sending a HEAD request to get the content length. Once you get the
content
length you create ranges and begin downloading in sections. All of this is
currently
available through the WebRequest feature set, except that you have to set the
headers
manually. I suggest grabbing the HTTP 1.1 RFC and going through that so you
fully
understand how ranges work. There are several different formats.

Currently, a large number of web servers support content range functionality,
since a good
deal of servers are running HTTP 1.1.
 
Most of what i got was 'how to write a successful resume', but I get the
picture - you're telling me it's completely impossible, without server
cooperation, right?
I can see why IE might not have implemented it - laziness, but if it was
possible Mozilla would have already done it wouldn't they? Damn!
Click to expand...

You need the FTP specifications

I doubt that it is part of HTTP
 
IE has supported resumed HTTP transfers for quite a while (I think, since IE
4or even 3), but it seems very picky about it. In many cases it won't
resume, just to be on a safe side. I think it happens when it cannot match
file timestamps in the original session and the resumed session.

If you ise Win32 Internet API, you can specify Range= field in the header,
to retrieve a part of file, if the server supports it.
 
I think it probably tries to implement the same http headers thing that
others have claimed IE does, but the BITS sample seems to work reasonably
well on the couple of tests I've done, and since I haven't got the foggiest
how the http headers thing works I don't want to spend ages learning how to
do something that I might be duplicating something that is already there but
doesn't work anyway.
Try the BITS sample, it's quite good...
 
Back
Top