Speeding up iterating through the filesystem on a file server.

  • Thread starter Thread starter Peter Strøiman
  • Start date Start date
P

Peter Strøiman

Hi.

I have an application that iterates through the file system of a file
server.
The application uses the classes DirectoryInfo and FileInfo to iterate
through
e.g.
void iterate( DirectoryInfo dir )
{
foreach ( DirectoryInfo subDir in dir.GetDirectories() )
{
iterate( subDir );
}
foreach ( FileInfo file in dir.GetFiles() )
{
doSomething ( file );
}
}

This takes quite a bit of time, when I have approx. 4Gb of data (many small
files)
Any ideas how I can speed it up?
I'm not an expert on code permission and security attributes. Are there any
attributes I can set in my program that would bypass code permission checks
( and only make them when starting up the app? )

Or are there quicker ways of obtaining the list of files and directories on
the server?

Thanks in advance,
Peter Strøiman
 
This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down. I usually
use VB.NET instead of C# so I'm not positive about the syntax but I think if
you do something like below, it will really speed things up. Let me know.

-Christine

int i;
int j;
DirectoryInfo[] diArr = di.GetDirectories();
int u = diArr.length -1;
DirectoryInfo subDir;
String fileArr[] = di.GetFiles;
Int k = fileArr.length - 1;
FileInfo file;

for(i=0; i <=u; i++){
subDir = diArr(i);
iterate(subDir);
for(j=0;j <= k; j++){
file = fileArr(j);
doSomething(file);
}
}

-Christine
 
Christine Nguyen said:
This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down.
Click to expand...

No it doesn't. It executes GetDirectories once, then gets the
enumerator from that.
 
Thanks for the reply.

First of all.

1: What I ( really ) do, is actually getting the array before iterating. I
simplified my example code.
2: It wouldn't call GetDirectories() for every iteration. Using the
foreach() syntax, the code will call GetDirectories() once and cast the
result to an IEnumerable.

Regards,
Peter Strøiman

Christine Nguyen said:
This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down. I usually
use VB.NET instead of C# so I'm not positive about the syntax but I think if
you do something like below, it will really speed things up. Let me know.

-Christine

int i;
int j;
DirectoryInfo[] diArr = di.GetDirectories();
int u = diArr.length -1;
DirectoryInfo subDir;
String fileArr[] = di.GetFiles;
Int k = fileArr.length - 1;
FileInfo file;

for(i=0; i <=u; i++){
subDir = diArr(i);
iterate(subDir);
for(j=0;j <= k; j++){
file = fileArr(j);
doSomething(file);
}
}

-Christine


Peter Strøiman said:
Hi.

I have an application that iterates through the file system of a file
server.
The application uses the classes DirectoryInfo and FileInfo to iterate
through
e.g.
void iterate( DirectoryInfo dir )
{
foreach ( DirectoryInfo subDir in dir.GetDirectories() )
{
iterate( subDir );
}
foreach ( FileInfo file in dir.GetFiles() )
{
doSomething ( file );
}
}

This takes quite a bit of time, when I have approx. 4Gb of data (many small
files)
Any ideas how I can speed it up?
I'm not an expert on code permission and security attributes. Are there any
attributes I can set in my program that would bypass code permission checks
( and only make them when starting up the app? )

Or are there quicker ways of obtaining the list of files and directories on
the server?

Thanks in advance,
Peter Strøiman
Click to expand...
Click to expand...
 
You're wrong. Calling for-each is allowed if the class implements
IEnumerable, witch has an method GetEnumerator() : IEnumerator. This
function is executed only once, when you enter the for-each. The IEnumerator
includes a state - machine for the loop.
You can speed up the code using an Index-Server Query.

GP

Christine Nguyen said:
This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down. I usually
use VB.NET instead of C# so I'm not positive about the syntax but I think if
you do something like below, it will really speed things up. Let me know.

-Christine

int i;
int j;
DirectoryInfo[] diArr = di.GetDirectories();
int u = diArr.length -1;
DirectoryInfo subDir;
String fileArr[] = di.GetFiles;
Int k = fileArr.length - 1;
FileInfo file;

for(i=0; i <=u; i++){
subDir = diArr(i);
iterate(subDir);
for(j=0;j <= k; j++){
file = fileArr(j);
doSomething(file);
}
}

-Christine


Peter Strøiman said:
Hi.

I have an application that iterates through the file system of a file
server.
The application uses the classes DirectoryInfo and FileInfo to iterate
through
e.g.
void iterate( DirectoryInfo dir )
{
foreach ( DirectoryInfo subDir in dir.GetDirectories() )
{
iterate( subDir );
}
foreach ( FileInfo file in dir.GetFiles() )
{
doSomething ( file );
}
}

This takes quite a bit of time, when I have approx. 4Gb of data (many small
files)
Any ideas how I can speed it up?
I'm not an expert on code permission and security attributes. Are there any
attributes I can set in my program that would bypass code permission checks
( and only make them when starting up the app? )

Or are there quicker ways of obtaining the list of files and directories on
the server?

Thanks in advance,
Peter Strøiman
Click to expand...
Click to expand...
 
All of you brought up really good points. I had forgotten about IEnumerator.
I've been able in the past to speed things up using this technique over
foreach, but maybe it was something particular to the collection I was
iterating.


Christine Nguyen said:
This is just a guess.

I think you can speed things up by using 2 standard for loops instead of
foreach. If you use "foreach ( DirectoryInfo subDir in
dir.GetDirectories() )", your code has to perform a GetDirectories every
time it iterates through the loop, and this really slows it down. I usually
use VB.NET instead of C# so I'm not positive about the syntax but I think if
you do something like below, it will really speed things up. Let me know.

-Christine

int i;
int j;
DirectoryInfo[] diArr = di.GetDirectories();
int u = diArr.length -1;
DirectoryInfo subDir;
String fileArr[] = di.GetFiles;
Int k = fileArr.length - 1;
FileInfo file;

for(i=0; i <=u; i++){
subDir = diArr(i);
iterate(subDir);
for(j=0;j <= k; j++){
file = fileArr(j);
doSomething(file);
}
}

-Christine


Peter Strøiman said:
Hi.

I have an application that iterates through the file system of a file
server.
The application uses the classes DirectoryInfo and FileInfo to iterate
through
e.g.
void iterate( DirectoryInfo dir )
{
foreach ( DirectoryInfo subDir in dir.GetDirectories() )
{
iterate( subDir );
}
foreach ( FileInfo file in dir.GetFiles() )
{
doSomething ( file );
}
}

This takes quite a bit of time, when I have approx. 4Gb of data (many small
files)
Any ideas how I can speed it up?
I'm not an expert on code permission and security attributes. Are there any
attributes I can set in my program that would bypass code permission checks
( and only make them when starting up the app? )

Or are there quicker ways of obtaining the list of files and directories on
the server?

Thanks in advance,
Peter Strøiman
Click to expand...
Click to expand...
 
Back
Top