Allocate more memory into Byte array

  • Thread starter Thread starter Arthur Yousif
  • Start date Start date
A

Arthur Yousif

Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long time on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBug = new Byte[1024];
iCharsRead = strResponse.Read(buf, 0, iChunk);
 
Arthur,

The GetResponseStream method on the HttpWebResponse is a class that
derives from Stream. You should be able to call the Read method on that
Stream and then pass how many bytes you want to read as well as the array
you want to read them into.

Hope this helps.
 
Hi Nicholas,

Man that was quick. :) I inadvertantly hit the wrong keys and my post went
before I finished it. I reposted it completed this time.

Here's my code:
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(m_drillLink);
wr.Timeout = Timeout.Infinite;

HttpWebResponse webResponse = (HttpWebResponse)wr.GetResponse();
System.IO.Stream strResponse = webResponse.GetResponseStream();


When I try to access the Length propety of strResponse, the .NET Framework
throws an exception because the CanSeek is false so I can't get that info.
Even with the Read method, I'd have to know the number of total bytes don't
I? I need to use a Byte[] array and must initialize it to some value, i.e.,
Byte[] buf = new Byte[1024]

Can I dynamically reallocate?

--
Arthur


Nicholas Paldino said:
Arthur,

The GetResponseStream method on the HttpWebResponse is a class that
derives from Stream. You should be able to call the Read method on that
Stream and then pass how many bytes you want to read as well as the array
you want to read them into.

Hope this helps.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Arthur Yousif said:
Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long
time
on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBug = new Byte[1024];
iCharsRead = strResponse.Read(buf, 0, iChunk);
 
MemoryStream mem=new MemoryStream();
byte[] buffer=new byte[1024];
int read=0;
do
{
read=instream.Read(buffer,0,buffer.Length);
mem.Write(buffer,0,read);
}while(read>0);

mem.Position=0;
StreamReader reader=new StreamReader(mem,true);
string contents=reader.ReadToEnd();

/Hugo
 
Hugo,

Thanks for the code, but I can't use a string. The stream must stay intact
as a binary stream. I had used the StreamReader but couldn't use its
methods. I'm loading a PDF binary stream.

However, your method of using a MemoryStream may work. I'll give that a
try. Thanks much.

--
Arthur


Hugo Wetterberg said:
MemoryStream mem=new MemoryStream();
byte[] buffer=new byte[1024];
int read=0;
do
{
read=instream.Read(buffer,0,buffer.Length);
mem.Write(buffer,0,read);
}while(read>0);

mem.Position=0;
StreamReader reader=new StreamReader(mem,true);
string contents=reader.ReadToEnd();

/Hugo

Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long time on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBug = new Byte[1024];
iCharsRead = strResponse.Read(buf, 0, iChunk);
 
Arthur,

Check HugoWetterberg's response to your post. Basically, he shows how
you can read the response in 1Kb chunks. What you can do is basically store
those in an ArrayList on each iteration, and then manipulate it however you
wish.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Arthur Yousif said:
Hi Nicholas,

Man that was quick. :) I inadvertantly hit the wrong keys and my post went
before I finished it. I reposted it completed this time.

Here's my code:
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(m_drillLink);
wr.Timeout = Timeout.Infinite;

HttpWebResponse webResponse = (HttpWebResponse)wr.GetResponse();
System.IO.Stream strResponse = webResponse.GetResponseStream();


When I try to access the Length propety of strResponse, the .NET Framework
throws an exception because the CanSeek is false so I can't get that info.
Even with the Read method, I'd have to know the number of total bytes don't
I? I need to use a Byte[] array and must initialize it to some value, i.e.,
Byte[] buf = new Byte[1024]

Can I dynamically reallocate?

--
Arthur


message news:[email protected]...
Arthur,

The GetResponseStream method on the HttpWebResponse is a class that
derives from Stream. You should be able to call the Read method on that
Stream and then pass how many bytes you want to read as well as the array
you want to read them into.

Hope this helps.


--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

Arthur Yousif said:
Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long
time
on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBug = new Byte[1024];
iCharsRead = strResponse.Read(buf, 0, iChunk);
 
One more question. Will the memory stream "break" if you write more than
the 1024 bytes you have allocated?

--
Arthur


Hugo Wetterberg said:
MemoryStream mem=new MemoryStream();
byte[] buffer=new byte[1024];
int read=0;
do
{
read=instream.Read(buffer,0,buffer.Length);
mem.Write(buffer,0,read);
}while(read>0);

mem.Position=0;
StreamReader reader=new StreamReader(mem,true);
string contents=reader.ReadToEnd();

/Hugo

Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long time on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBug = new Byte[1024];
iCharsRead = strResponse.Read(buf, 0, iChunk);
 
It isn't a string it is a binary memory stream. I just used a stream
reader to show how a memory stream worked. I would really reccomend
the memory stream over som hack with buffers in a array list.
/Hugo
 
Hugo said:
It isn't a string it is a binary memory stream. I just used a stream
reader to show how a memory stream worked. I would really reccomend
the memory stream over som hack with buffers in a array list.

Yes, a much cleaner solution.
 
I feel kinda' stupid asking this, but doesn't the documentation say it
implements a TextReader? Also I had tried it in the past and it didn't work
because the binary characters were being changed to characters.

I was green when I started using this stuff 6 months ago, so I need to
revisit some of this stuff. I was "under the gun" to get stuff out at the
time. BTW, here's the solution I've used for now, and you're right, it's a
hack, but it works for now. :)

Thanks again for your help. I really appreciate it.

// create a web request to pass the drill link to. We'll set
// the timeout to what we have in the web.config file
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(m_drillLink);

// the web request object's Timeout property expects the number
// in milliseconds
int iTimeoutInMinutes =
Convert.ToInt32(ConfigurationSettings.AppSettings["dlTimeout"]);
wr.Timeout = (1000 * 60) * iTimeoutInMinutes;

// create a response object so we can retrieve the stream. The stream
// will let us read the response from the URL we passed.
HttpWebResponse webResponse = (HttpWebResponse)wr.GetResponse();
System.IO.Stream strm = webResponse.GetResponseStream();

// create an array list that will hold the binary bytes
// in chunks. Each chunk will then be read back and assembled
// into one byte binary array. We have to do it this way
// because the stream object doesn't know the length of the
// stream, so we have to determine it dynamically.
ArrayList arBytes = new ArrayList();

// we'll use 1024 size chunks to read the data stream.
// we'll keep track of the total number of bytes read
// and how many characters are read with each request.
int iChunk = 1024, iCharsRead = 0;
long lTotalBytesRead = 0;

// create a hold buffer we'll put the data into. We'll
// initialize using the chunk size.
Byte[] bufHold = new Byte[iChunk];

// make the initial read into the hold buffer
iCharsRead = strm.Read(bufHold, 0, iChunk);

// create a buffer using the actual # of chars read
// and copy the contents of the hold buffer into this
// one. We do this to make sure that the buffer we
// want to save into the array list always has the
// actual number of characters read.
Byte[] buf = new Byte[iCharsRead];
Array.Copy(bufHold, 0, buf, 0, iCharsRead);

// when no more chars are available to read, then the
// # of chars read will be 0, so let's loop until all
// the bytes have been read from the stream
while (iCharsRead != 0)
{
// keep track of the total bytes read to use later
// to reconstruct all the blocks into one Byte array
lTotalBytesRead += iCharsRead;

// add the current buffer block
arBytes.Add(buf);

// reinitialize the hold buffer and read the next
// block into it
bufHold = new Byte[iChunk];
iCharsRead = strm.Read(bufHold, 0, iChunk);

// reinitialize the store buffer using the actual #
// of bytes read and copy into it from the hold buffer
buf = new Byte[iCharsRead];
Array.Copy(bufHold, 0, buf, 0, iCharsRead);
}

// create the binary string we'll use to send the data
// to the user. We'll initialize it to the total # of
// bytes we've read from the stream.
Byte[] binStr = new Byte[lTotalBytesRead];

// set the index to 0 to start. This will be incremented
// as we read each block and store it starting at the
// index location pointed to by iIdx.
int iIdx = 0;

// loop through the Byte array list and process for each
// array list's Byte[] object.
foreach (Byte[] dataBuf in arBytes)
{
// copy the contents of dataBuf into binStr starting
// at the iIdx index location.
dataBuf.CopyTo(binStr, iIdx);

// increment the index location by the content Byte[]'s
// length to make sure the next block is written at the
// correct starting location.
iIdx += dataBuf.Length;
}

// Release the response object resources.
strm.Close();
webResponse.Close();
 
Arthur said:
I feel kinda' stupid asking this, but doesn't the documentation say it
implements a TextReader? Also I had tried it in the past and it didn't work
because the binary characters were being changed to characters.

A MemoryStream is just a stream. It can be passed to the constructor of
a StreamWriter, which then 'wraps' the MemoryStream in a TextWriter.
It's possible that this is what happened before.
I was green when I started using this stuff 6 months ago, so I need to
revisit some of this stuff. I was "under the gun" to get stuff out at the
time. BTW, here's the solution I've used for now, and you're right, it's a
hack, but it works for now. :)

Thanks again for your help. I really appreciate it.

// create a web request to pass the drill link to. We'll set
// the timeout to what we have in the web.config file
HttpWebRequest wr = (HttpWebRequest)WebRequest.Create(m_drillLink);

// the web request object's Timeout property expects the number
// in milliseconds
int iTimeoutInMinutes =
Convert.ToInt32(ConfigurationSettings.AppSettings["dlTimeout"]);
wr.Timeout = (1000 * 60) * iTimeoutInMinutes;

// create a response object so we can retrieve the stream. The stream
// will let us read the response from the URL we passed.
HttpWebResponse webResponse = (HttpWebResponse)wr.GetResponse();
System.IO.Stream strm = webResponse.GetResponseStream();

// create an array list that will hold the binary bytes
// in chunks. Each chunk will then be read back and assembled
// into one byte binary array. We have to do it this way
// because the stream object doesn't know the length of the
// stream, so we have to determine it dynamically.
ArrayList arBytes = new ArrayList();

// we'll use 1024 size chunks to read the data stream.
// we'll keep track of the total number of bytes read
// and how many characters are read with each request.
int iChunk = 1024, iCharsRead = 0;
long lTotalBytesRead = 0;

// create a hold buffer we'll put the data into. We'll
// initialize using the chunk size.
Byte[] bufHold = new Byte[iChunk];

// make the initial read into the hold buffer
iCharsRead = strm.Read(bufHold, 0, iChunk);

// create a buffer using the actual # of chars read
// and copy the contents of the hold buffer into this
// one. We do this to make sure that the buffer we
// want to save into the array list always has the
// actual number of characters read.
Byte[] buf = new Byte[iCharsRead];
Array.Copy(bufHold, 0, buf, 0, iCharsRead);

// when no more chars are available to read, then the
// # of chars read will be 0, so let's loop until all
// the bytes have been read from the stream
while (iCharsRead != 0)
{
// keep track of the total bytes read to use later
// to reconstruct all the blocks into one Byte array
lTotalBytesRead += iCharsRead;

// add the current buffer block
arBytes.Add(buf);

// reinitialize the hold buffer and read the next
// block into it
bufHold = new Byte[iChunk];
iCharsRead = strm.Read(bufHold, 0, iChunk);

// reinitialize the store buffer using the actual #
// of bytes read and copy into it from the hold buffer
buf = new Byte[iCharsRead];
Array.Copy(bufHold, 0, buf, 0, iCharsRead);
}

// create the binary string we'll use to send the data
// to the user. We'll initialize it to the total # of
// bytes we've read from the stream.
Byte[] binStr = new Byte[lTotalBytesRead];

// set the index to 0 to start. This will be incremented
// as we read each block and store it starting at the
// index location pointed to by iIdx.
int iIdx = 0;

// loop through the Byte array list and process for each
// array list's Byte[] object.
foreach (Byte[] dataBuf in arBytes)
{
// copy the contents of dataBuf into binStr starting
// at the iIdx index location.
dataBuf.CopyTo(binStr, iIdx);

// increment the index location by the content Byte[]'s
// length to make sure the next block is written at the
// correct starting location.
iIdx += dataBuf.Length;
}

// Release the response object resources.
strm.Close();
webResponse.Close();
 
Hey Hugo,

I wanted to let you know that I've implemented it as below, however, I do
remember now that I had tried it this way a long time ago and the reason I
couldn't get it to work is because I was using the StreamReader.

In my case, all I really needed was the raw binary data so here's what I do
after repositioning the pointer:

mem.Position = 0;


arBin = new Byte[mem.Length];
mem.Read(arBin, 0, (int)mem.Length);

Thanks again.

--
Arthur


Hugo Wetterberg said:
MemoryStream mem=new MemoryStream();
byte[] buffer=new byte[1024];
int read=0;
do
{
read=instream.Read(buffer,0,buffer.Length);
mem.Write(buffer,0,read);
}while(read>0);

mem.Position=0;
StreamReader reader=new StreamReader(mem,true);
string contents=reader.ReadToEnd();

/Hugo

Hello,

I have a stream I'm reading from, which is from an HttpWebRequest and the
Length property is not available because the stream doesn't support seek
operations.

I'm looping through the stream byte by byte but that's taking a long time on
really big files. It would be nice to do something like I do in my C++
applications where I read the data in chunks and then reallocation more
memory to my destination block.

Is there a way to do something similar in ASP.NET using C#? Something like:

Byte[] destBug = new Byte[1024];
iCharsRead = strResponse.Read(buf, 0, iChunk);
 
Back
Top