C
Charles
Yes, I agree. I think it will make sense for it to be a thread; we have
enough separate processes running already.
Back to the end of my original question, I suppose it is too much to hope
that there is a tried and tested mechanism to handle all this for me?
Charles
enough separate processes running already.
Back to the end of my original question, I suppose it is too much to hope
that there is a tried and tested mechanism to handle all this for me?
Charles
r norman said:If the data must be loaded into the database in chronological order,
then you have no choice. ALL data must be stuffed into a queue as it
arrives. A separate thread (process?) then attempts to store it all
into the database as fast as the database will accept it. If the
database is unavailable for a time, the queue just backs up but all
new data still goes to the back of the line.
Even if you load the files every 30 seconds or every minute?
Hmm. Maybe.
The tricky bit, as I see it, is how to manage the data load when the
database comes back on-line. The incoming data stream is relentless, and
there is no way of stopping it. When the database goes off-line, we can
certainly divert the stream to a file. When the database comes back, the
file has to be emptied to the database whilst data are still coming in.
Managing that last bit so that the data are loaded in chronological order
is
where I see the complexity.
Charles
message
Even if you load the files every 30 seconds or every minute?
Well you could certainly have a flag that the app looks at that says
"write
to DB" or "write to file"... When the flag is set to the latter, it
writes
to the file instead. When the database comes back up, you change the
flag
back, and load any files that were written in the meantime.
On 5/23/09 2:14 PM, in article (e-mail address removed),
Hi Aaron
Thanks for the quick reply. The data need to be pushed to the database
because it is a [near] realtime system. When the database is up, the
data
need to be saved in the database immediately. The times when it is not
possible to reach the database are very rare, but we need to design for
that
eventuality so that we don't lose data on those rare occasions.
... including turning off the process when
you want to perform maintenance on the database
This would be a very attractive feature when we have the solution in
place,
but, as I say, when the database is up the data need to be saved as it
arrives, and not on batch.
Charles