Allocate more memory into Byte array

  • Thread starter Thread starter Arthur Yousif
  • Start date Start date
A

Arthur Yousif

Sorry, my previous post was posted by accident before I finished it.

Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long time on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBuf = new Byte[1024];
int iCharsRead = strResponse.Read(destBuf, 0, 1024);
int idx = 0;
while (iCharsRead != 0)
{
idx += iCharsRead;
lTotalBytesRead += iCharsRead;

if (iCharsRead < iChunk)
break;

// here I would allocate more memory space to destBuf and then
// read the response stream's data into the destBuf at the idx location
// but I can't find a way to do it in c#. Any ideas?

iCharsRead = strResponse.Read(destBuf, idx, 1024);
}
 
Arthur said:
Sorry, my previous post was posted by accident before I finished it.

Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long time on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBuf = new Byte[1024];
int iCharsRead = strResponse.Read(destBuf, 0, 1024);
int idx = 0;
while (iCharsRead != 0)
{
idx += iCharsRead;
lTotalBytesRead += iCharsRead;

if (iCharsRead < iChunk)
break;

// here I would allocate more memory space to destBuf and then
// read the response stream's data into the destBuf at the idx location
// but I can't find a way to do it in c#. Any ideas?

iCharsRead = strResponse.Read(destBuf, idx, 1024);
}
Click to expand...

you can't reallocate an array as a single operation in C#. One thing
you can do is allocate a new array, then copy over the existing one:

byte [] temp = new byte[ idx + 1024];
destBuf.CopyTo( temp, 0);
destBuf = temp;

However, that means that for each pass through the loop, you're copying
the entire array over again. Something you might want to consider is
simply storing each buffer in an ArrayList, then after everything is
read, you can copy the data over into one big array once.

Something like the following (which I have not compiled and tested, so
it may need some tweaking):

const int bufsize = 1024;

ArrayList bufList = new ArrayList();
Byte[] destBuf = new Byte[bufsize];
int iCharsRead = strResponse.Read(destBuf, 0, bufsize);
int idx = 0;
while (iCharsRead != 0)
{
idx += iCharsRead;
lTotalBytesRead += iCharsRead;

if (iCharsRead < bufsize) {
// if we have a partial read (ie., we're at the end of the
// stream, bail out before adding destBuf to the list
break;
}

bufList.Add( destBuf);

destBuf = new byte [bufsize];

iCharsRead = strResponse.Read(destBuf, 0, bufsize);
}

// now copy the data into one big array
byte [] finalBuf = new byte [idx];

int i = 0;
foreach (byte [] b in bufList) {
b.CopyTo( finalBuf, i);
i += bufsize;
}
destBuf.CopyTo( finalBuf, i);
 
Great minds think alike. ;)

That's exactly what I'm doing now. Thanks a lot Mike and everyone else for
your help.

--
Arthur


mikeb said:
Arthur said:
Sorry, my previous post was posted by accident before I finished it.

Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long time on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBuf = new Byte[1024];
int iCharsRead = strResponse.Read(destBuf, 0, 1024);
int idx = 0;
while (iCharsRead != 0)
{
idx += iCharsRead;
lTotalBytesRead += iCharsRead;

if (iCharsRead < iChunk)
break;

// here I would allocate more memory space to destBuf and then
// read the response stream's data into the destBuf at the idx location
// but I can't find a way to do it in c#. Any ideas?

iCharsRead = strResponse.Read(destBuf, idx, 1024);
}
Click to expand...

you can't reallocate an array as a single operation in C#. One thing
you can do is allocate a new array, then copy over the existing one:

byte [] temp = new byte[ idx + 1024];
destBuf.CopyTo( temp, 0);
destBuf = temp;

However, that means that for each pass through the loop, you're copying
the entire array over again. Something you might want to consider is
simply storing each buffer in an ArrayList, then after everything is
read, you can copy the data over into one big array once.

Something like the following (which I have not compiled and tested, so
it may need some tweaking):

const int bufsize = 1024;

ArrayList bufList = new ArrayList();
Byte[] destBuf = new Byte[bufsize];
int iCharsRead = strResponse.Read(destBuf, 0, bufsize);
int idx = 0;
while (iCharsRead != 0)
{
idx += iCharsRead;
lTotalBytesRead += iCharsRead;

if (iCharsRead < bufsize) {
// if we have a partial read (ie., we're at the end of the
// stream, bail out before adding destBuf to the list
break;
}

bufList.Add( destBuf);

destBuf = new byte [bufsize];

iCharsRead = strResponse.Read(destBuf, 0, bufsize);
}

// now copy the data into one big array
byte [] finalBuf = new byte [idx];

int i = 0;
foreach (byte [] b in bufList) {
b.CopyTo( finalBuf, i);
i += bufsize;
}
destBuf.CopyTo( finalBuf, i);
Click to expand...
 
Arthur said:
Sorry, my previous post was posted by accident before I finished it.

Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long time on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBuf = new Byte[1024];
int iCharsRead = strResponse.Read(destBuf, 0, 1024);
int idx = 0;
while (iCharsRead != 0)
{
idx += iCharsRead;
lTotalBytesRead += iCharsRead;

if (iCharsRead < iChunk)
break;

// here I would allocate more memory space to destBuf and then
// read the response stream's data into the destBuf at the idx location
// but I can't find a way to do it in c#. Any ideas?

iCharsRead = strResponse.Read(destBuf, idx, 1024);
}
Click to expand...

One other thing you can do to possibly optimize certain instances of
this is to check the ContentLength property of the web request. If it's
set, you should be able to use that information to allocate the correct
size buffer, and read everything right into it. However, you can't
depend on it being set, so you'd have to be able to fall back to a
variation of what you're proposing now as well.
 
actually the stream reader is already buffering in chucks, and you would be
just adding more layers.

for a test, compare single char/byte reads (no processing) to the ReadToEnd
method, and you should see no difference in timings.

-- bruce (sqlwork.com)
 
Back
Top