G
Guest
Background:
We are currently working on an ETL project, moving tons of data into
multiple tables in a database. Due to the amount of work, the process bogs
down the database server when the work is done from feed tables on the
database.
Process:
To speed up the process and not hammer the database server, we are using a
variety of offline processes to filter, sort and remove dupes. The result of
the process are BCP files that get bulk loaded into the database.
Issue:
Today, the process was indicating failure on a test run. Looking at the
tables, everything was fine, but the object that wrapped BCP, using
System.Diagnostics. Process, was throwing an exception.
Solution:
The code was like this:
try
{
BCPProcess.Start();
startime = BCPProcess.StartTime;
}
catch (Exception ex)
{
//More here
}
Although we will be working with millions of rows ultimately, the test files
are 1,000 records long. BCP, even on a desktop, can move more than 10,000
lines per second with 50+ columns. Working with a 1,000 line, single column
file, the process finished before Start() returned, causing the call to
BCPProcess.StartTime to throw an exception.
While not documented in the help file, if the process completes before you
query anything (with the exception of Standard Output), you will throw an
exception, as the process is no longer there to query for information. Very
clever.
--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA
***************************
Think Outside the Box!
***************************
We are currently working on an ETL project, moving tons of data into
multiple tables in a database. Due to the amount of work, the process bogs
down the database server when the work is done from feed tables on the
database.
Process:
To speed up the process and not hammer the database server, we are using a
variety of offline processes to filter, sort and remove dupes. The result of
the process are BCP files that get bulk loaded into the database.
Issue:
Today, the process was indicating failure on a test run. Looking at the
tables, everything was fine, but the object that wrapped BCP, using
System.Diagnostics. Process, was throwing an exception.
Solution:
The code was like this:
try
{
BCPProcess.Start();
startime = BCPProcess.StartTime;
}
catch (Exception ex)
{
//More here
}
Although we will be working with millions of rows ultimately, the test files
are 1,000 records long. BCP, even on a desktop, can move more than 10,000
lines per second with 50+ columns. Working with a 1,000 line, single column
file, the process finished before Start() returned, causing the call to
BCPProcess.StartTime to throw an exception.
While not documented in the help file, if the process completes before you
query anything (with the exception of Standard Output), you will throw an
exception, as the process is no longer there to query for information. Very
clever.
--
Gregory A. Beamer
MVP; MCP: +I, SE, SD, DBA
***************************
Think Outside the Box!
***************************