Why is serialization so SLOW?

  • Thread starter Thread starter Mountain Bikn' Guy
  • Start date Start date
M

Mountain Bikn' Guy

We have an app that computes a lot of numeric data. We would like to save to
disk about 1-2 gigabytes of computed data that includes ints, doubles,
strings and some complex objects that contain hashtables. We would like to
read this data back into the app and reuse those values and state to compute
more new data.

Up to this point we just write out comma separated ASCII values for
everything. And we read/parse the ascii back in. The ASCII input/output code
isn't very optimized, but it works. We would like to gain greater reuse and
increased performance by writing out data in binary format.

We did some testing and careful design, and implemented what we thought was
good quality binary output code. We used binary formatters with
serialization. The code is more carefully designed and tested than our
previous ASCII output code. However, at best, it is 14 times slower! It is
shockingly slow compared to something that we didn't think was that fast to
start with.

We've seen a lot of people posting similar experiences with .NET
serialization being slow. We also use it in our app for saving various
settings and enabling undo/redo. In those circumstances, it is also very
slow.

It seem slowest when dealing with complex objects. In our testing with just
doubles and strings, serialization in binary format was faster than ASCII
with parsing. But in the real life app with real data, serialization just
doesn't cut it.

Any tips or suggestions are appreciated. Is there another approach to
persisting complex program states to disk?
 
Hello

Serialization can store much overhead data, like data types, member field
names, etc.

Try using System.IO.BinaryReader and System.IO.BinaryWriter classes,
although it is much harder than serialization, but it will outperform it.
You will have to implement saving a hashtable in binary format

The way you store you information is also important, with 1-2 GB of info,
you must have a way to index the data, so that you can get the data you need
without searching the whole file.

Also, have you considered storing the data in a database server, like SQL
server?

Best Regards

Sherif
 
Back
Top