Bulk insert into Database

  • Thread starter Thread starter Baz
  • Start date Start date
B

Baz

Hi,

I am designing an application that needs to do the following on a daily
basis:
- Parse XML files & extract relevant information. The number of XML files is
in thousands...
- Write the extracted information to a database.

I was considering the use of Datasets for this application. However, aren't
Datasets more suited in situations where you select (query) some data from
the database & perform updates on it? Would Datasets be efficient in cases
where there is only one-way communication between the client & database
server, i.e., data only travels from the client to server & only bulk
inserts are performed.

If Datasets aren't the best way to approach this, could I get some
suggestions on how to design this? Would it be more efficient to extract
records into a flat file & write those to a database using a stored
procedure?

Any suggestions welcome.
thanks!
 
Baz:

Are you using SQL Server? This is probably well suited to DTS or BCP. You
can do it with dataset, but at present, ADO.NET is a dataaccess technology
that's not really intended to bulk transfers. DataSet's contain a bunch of
information (ie diffgram) that tracks rowstate so that the dataadapter knows
what it'll need to do , so yes, it's much better suited to update scenarios.
BCP is probably the fastest way to get data around so i'd go for it if
possible

HTH,

Bill
 
Thanks for the reply Bill. Your post confirms what I suspected.

We might end up using Oracle. Does Oracle have a command similar to bcp?

Baz
 
Baz:

I haven't used Oracle since 8i but I believe SQL Loader is the equivalent.
I can't imagine that it doesn't have something that's essentially the same.
 
Is there a command similar to bcp for virtually any datasource?

I dont know if you saw my post but I was hoping to find out how to
bulk insert data from ANY datasource.

Is there a solution to this that can be used for any datasource
(oracle, sql server, text, access, etc.) but is more efficient than
looping + insert?

thanks for your help.
 
Back
Top