Fastest way of accessing CSV files

  • Thread starter Thread starter Shiplu
  • Start date Start date
S

Shiplu

hi,
I am handling CSV files as database. each CSV got 40,000 - 60,000
lines. I have to do the joining, subquery and other complex tasks here.
the problem is, my query execution time too high, 2-3 mins. how can I
reduce the time ? I saw there is no way of declaring keys for CSV files
in Schema.ini. I am using OleDb.
anotherthing,
can anyone tell me what should I use to this heavy task ? is there any
good replacement of OleDb ?
please let me know.
 
Do you have access to any version of SQL Server?

What I learned to do was simply import data from a CSV file right into
a SQL table, that way I can handle data processing via T-SQL and the
SQL Engine.
 
Another poster gave you half of the answer--import into a SQL Server table
using SqlBulkCopy.

--
____________________________________
William (Bill) Vaughn
Author, Mentor, Consultant
Microsoft MVP
INETA Speaker
www.betav.com/blog/billva
www.betav.com
Please reply only to the newsgroup so that others can benefit.
This posting is provided "AS IS" with no warranties, and confers no rights.
__________________________________
 
Atually I am not using any SQL server.
my database is just a folder and tables are some files
I am using C# .NET
any Idea ?
 
Well, I don't see how you'll be able to gain much speed if you limit
yourself to keeping your data in large, unindexable text files. You
could try using the MS Jet as the query engine, unless that is what
you're already doing, to see if that could help at all.

SQL Express is free to download and install. You could even try the
SQL Everywhere engine, which is sorta like Jet in that it's just a set
of LIBs that load at runtime by the client app, instead of a service.

Hope this helps.
 
Back
Top