How to write array of doubles to stream without using a loop?

  • Thread starter Thread starter John Dumais
  • Start date Start date
J

John Dumais

Hello,

I have been trying to figure out how to write an array of doubles (in
this specific case) to a binary stream without using a loop. What I
have been doing is...

foreach(double d in TraceData)
{
instanceOfBinaryWriter.Write(d);
}

The loop is introducing overhead I don't want. In unmanaged c++, I
would just do something like...

fwrite(buffer, sizeof(double), numArrayElements, fileStream)

Is their an equivalent using c#?

Thanks,
 
Surely fwrite would have a loop inside it? If it does, then surely it
doesn't matter if the loop occurs in your code or in library code?
 
John,

You won't really be able to do that. It would be easier if you had
direct memory access, but unfortunately, you don't. The only thing that you
could do is do a conversion to a byte array, but you know that will
introduce overhead as well (that routine has to loop through as well to do
the conversion).

The only other option you would have is if your method can use unsafe
code. That way, you could cast the double array to a double pointer, and
then cast that to a pointer of a byte, and then pass the byte to the method
without having to loop through to get a byte array.

Hope this helps.
 
You won't really be able to do that. It would be easier if you had
direct memory access, but unfortunately, you don't. The only thing that you
could do is do a conversion to a byte array, but you know that will
introduce overhead as well (that routine has to loop through as well to do
the conversion).
Click to expand...

Take a look at Buffer.BlockCopy. It only works with primitive types. But I
remember looking at the rotor source code and its an internal call and does
a memcopy , memove, mem something but not a for loop. The docs say "This
class provides better performance for manipulating primitive types than
similar methods in the System.Array class.". Here is a sample that did work.

Dim Ds() As Integer = {1, 2, 3}
Dim I() As Byte = Array.CreateInstance(GetType(Byte), Buffer.ByteLength(Ds))
Buffer.BlockCopy(Ds, 0, I, 0, Buffer.ByteLength(Ds))



Nicholas Paldino said:
John,

You won't really be able to do that. It would be easier if you had
direct memory access, but unfortunately, you don't. The only thing that you
could do is do a conversion to a byte array, but you know that will
introduce overhead as well (that routine has to loop through as well to do
the conversion).

The only other option you would have is if your method can use unsafe
code. That way, you could cast the double array to a double pointer, and
then cast that to a pointer of a byte, and then pass the byte to the method
without having to loop through to get a byte array.

Hope this helps.

--
- Nicholas Paldino [.NET/C# MVP]
- (e-mail address removed)

John Dumais said:
Hello,

I have been trying to figure out how to write an array of doubles (in this
specific case) to a binary stream without using a loop. What I have been
doing is...

foreach(double d in TraceData)
{
instanceOfBinaryWriter.Write(d);
}

The loop is introducing overhead I don't want. In unmanaged c++, I would
just do something like...

fwrite(buffer, sizeof(double), numArrayElements, fileStream)

Is their an equivalent using c#?

Thanks,
Click to expand...
Click to expand...
 
John Dumais said:
I have been trying to figure out how to write an array of doubles (in
this specific case) to a binary stream without using a loop. What I
have been doing is...

foreach(double d in TraceData)
{
instanceOfBinaryWriter.Write(d);
}

The loop is introducing overhead I don't want.
Click to expand...

Have you measured the overhead? Is it definitely causing a problem?
There's a tendency to assume that things like this are bottlenecks
without experimentation - that may not be the case here, but it's worth
knowing before we get into much riskier code.
 
Jon said:
Have you measured the overhead? Is it definitely causing a problem?
There's a tendency to assume that things like this are bottlenecks
without experimentation - that may not be the case here, but it's worth
knowing before we get into much riskier code.
Click to expand...

Not specifically in C#, because I can't get an apples-to-apples
comparison, but my experience has been that most run-time
libraries work better when you supply them with one hunk of data
as opposed to supplying many little hunks in a loop. The attached
example will illustrate. It's in C++, because I can't come up
with a reasonable comarison in C#. Sorry in advance -- I wrote
this illustration on my Linux machine. I don't have a Windows
PC at home. I ran the program 10 times in a loop and got the
following results.
Using loop
Elapsed time: 2 seconds, 223613 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 725197 micro-seconds
Using loop
Elapsed time: 3 seconds, 256561 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 277398 micro-seconds
Using loop
Elapsed time: 2 seconds, 224807 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 45266 micro-seconds
Using loop
Elapsed time: 2 seconds, 215541 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 231558 micro-seconds
Using loop
Elapsed time: 2 seconds, 221532 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 278577 micro-seconds
Using loop
Elapsed time: 2 seconds, 220584 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 11936 micro-seconds
Using loop
Elapsed time: 2 seconds, 215263 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 28349 micro-seconds
Using loop
Elapsed time: 2 seconds, 223527 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 273473 micro-seconds
Using loop
Elapsed time: 2 seconds, 226312 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 7940 micro-seconds
Using loop
Elapsed time: 2 seconds, 217676 micro-seconds
Writing in a single hunk
Elapsed time: 1 seconds, 23728 micro-seconds
Click to expand...

The program...
#include <cstdio>
#include <cstdlib>
#include <sys/time.h>

typedef void(*WriterFuncPtr)(FILE*, double*, size_t);

void timeIt(WriterFuncPtr funcPtr, FILE *fp, double *data,
size_t numDataPoints)
{
timeval startTime = {0, 0};
timeval endTime = {0, 0};
timeval elapsedTime = {0, 0};

if( ! gettimeofday(&startTime, 0)){
funcPtr(fp, data, numDataPoints);

if( ! gettimeofday(&endTime, 0)){
if(startTime.tv_usec > endTime.tv_usec){
endTime.tv_usec += 1000000;
endTime.tv_sec--;
}

elapsedTime.tv_usec = endTime.tv_usec - startTime.tv_usec;
elapsedTime.tv_sec = endTime.tv_sec - startTime.tv_sec;

printf("Elapsed time: %ld seconds, %ld micro-seconds\n",
elapsedTime.tv_sec, elapsedTime.tv_usec);
}
}

}

void checkFileContents(FILE *fp)
{
double data = 0;

size_t numRead = 0;

do{
numRead = fread(&data, sizeof(data), 1, fp);
printf("%lf\n", data);
} while(numRead > 0);
}

void writeDataUsingLoop(FILE *fp, double *data, size_t numDataPoints)
{
for(size_t i = 0; i < numDataPoints; ++i){
(void)fwrite(data++, sizeof(double), 1, fp);
}
}

void writeDataInOneHunk(FILE *fp, double *data, size_t numDataPoints)
{
(void)fwrite(data, sizeof(double), numDataPoints, fp);
}

int main(void)
{
FILE *fp = fopen("data", "w");
if(fp){
const size_t numDataPoints = 8 * 1024 * 1024;
double *data = (double*)malloc(numDataPoints * sizeof(double));
if(data){
for(size_t i = 0; i < numDataPoints; ++i){
data = i;
}

printf("Using loop\n");
timeIt(writeDataUsingLoop, fp, data, numDataPoints);

fclose(fp);

fp = fopen("data", "w");
if(fp){
printf("Writing in a single hunk\n");
timeIt(writeDataInOneHunk, fp, data, numDataPoints);

fclose(fp);
fp = 0;
}

free(data);
}
}

/*
fp = fopen("data", "r");
if(fp){
checkFileContents(fp);

fclose(fp);
fp = 0;
}
*/

return 0;
}
Click to expand...


Thanks,
 
Back
Top