Http stream performance

  • Thread starter Thread starter Leon Jollans
  • Start date Start date
L

Leon Jollans

Hi.
I'm reading a 5 meg (or so) file from the response stream of a WebRequest,
and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to
download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage), but I can't have it taking this long to download in the
first instance. We've got Gigabit LAN. what's the bottleneck?

I could make the resource available via a different protocol / means if
necessary to overcome the problem.

Has anyone had similar issues or know of a better method of large network
downloads?

Cheers,
Leon
 
Leon Jollans said:
I'm reading a 5 meg (or so) file from the response stream of a WebRequest,
and it's pitifully slow. I mean *really* slow, in the order of 15 minutes to
download. Now obviously I'm caching this on the client (Windows app /
IsolatedStorage), but I can't have it taking this long to download in the
first instance. We've got Gigabit LAN. what's the bottleneck?
Click to expand...

How are you reading it, exactly? (Please show some code, in other
words.) Have you tried reading it with a plain browser (or wget) to see
whether it's the client or the server that's holding it up?
 
It was definitely the client all along. but I think I've sussed it now: I
was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.

now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();

...

cheers

Leon
 
Leon Jollans said:
It was definitely the client all along. but I think I've sussed it now: I
was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.
Click to expand...

Yes, that would definitely slow it down :)
now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();
Click to expand...

Rather than doing that, I'd do something like:
(assuming you can make contentLength an int instead of a long)


byte[] data = new byte[contentLength];

int offset=0;
int remaining = contentLength;

while (remaining > 0)
{
int read = stream.Read (data, offset, remaining);
if (read==-1)
// Up to you - throw an exception or something similar
// because we didn't get as much as we expected
offset+=read;
remaining-=read;
}
 
That's much faster! thanks :)

Leon

Jon Skeet said:
Leon Jollans said:
It was definitely the client all along. but I think I've sussed it now: I
was reading the stream byte by byte into a pre-initialised array using
ReadByte from the stream directly and checking for -1 each time.
Click to expand...

Yes, that would definitely slow it down :)
now I'm doing it using a BinaryReader in chunks and the speed is as per
expectations. code below for reference.

void readStream(Stream stream, long contentLength){
int i=0;
byte[] bytes;
BinaryReader reader = new BinaryReader(stream);
MemoryStream ms = new MemoryStream((int)contentLength);
do
{
bytes = reader.ReadBytes(128);
if(bytes.Length > 0)
{
ms.Write(bytes,0,bytes.Length);
}
else{
break;
}
}while( true );
reader.Close();
stream.Close();
ms.Flush();
bytes = ms.ToArray();
ms.Close();
Click to expand...

Rather than doing that, I'd do something like:
(assuming you can make contentLength an int instead of a long)


byte[] data = new byte[contentLength];

int offset=0;
int remaining = contentLength;

while (remaining > 0)
{
int read = stream.Read (data, offset, remaining);
if (read==-1)
// Up to you - throw an exception or something similar
// because we didn't get as much as we expected
offset+=read;
remaining-=read;
}
Click to expand...
 
Back
Top