more then 105000 folders in one folder

  • Thread starter Thread starter Marco J.L.
  • Start date Start date
M

Marco J.L.

Hello,

we have an performance issue, sometimes our server has a lack of
performance.
Operating System Windows 2000 and Windows 2003 (both give the performance
problemens)
When opening the Explorer and only the drive which contains the files (not
the folder) the explorer is using 50 - 90 % of the CPU and the memory is
fluctuating between 60 and 110 kb, it fluctuates continouisly.

I think that the problem is caused by this :
We've an application which uses in ONE single folder more then 105.000
folders, each folder contains ONE single file.

I told it to our programmers but they don't see any problems with it. I've
some questions :
** 1 ** Has Microsoft a recommendation about the maximum number of files in
one folder?
** 2 ** Which performance counters are the best to use to monitor if this is
a problem?

thank you in advance,

Marco J.L.
(e-mail address removed)
 
Our application experienced something similar to your problem. Not sure if
our application is entirely related to how you write files, but this may
help. We may store over 4,000,000 files per folder using multiple folders.
The performance on our system would degrade over time. Usually when the
drive was about 60% full.

We opened a call with Microsoft about the same performance issues that you
described. Our system performance was going in the tank whenever we
starting getting a lot of files on the drive. NTFS is supposedly capable of
handling this, but nonetheless, the performance problems resulted. MS told
us that we may be causing a problem with over allocation of small clusters
and this could lead to the performance problems. In order to avoid the
problem, they suggested, when creating a file, to set the maximum size of
the file prior to writing the data. Believe it or not, this helped a lot.
Here is a brief rundown of their solution which corrected the problem:

PROBLEM
========
Disk subsystem degrades with writing millions of files

RESOLUTION
============
Open the file
Set the filesize to the expected size using SetEndOfFile.
Do the write

Rather than just
Open the file
Do the write


Hope this helps.



--
-----------------------------------
Ken Varn
Senior Software Engineer
Diebold Inc.

EmailID = varnk
Domain = Diebold.com
 
Back
Top