Reading the first file from a directory.

  • Thread starter Thread starter Craig Vermeer
  • Start date Start date
C

Craig Vermeer

Hi All,

I have a program that's using the file system as a queuing mechanism,
and it's consuming an inordinate amount of CPU time when the file system
queue gets all that large (any more than a thousand or so messages in
the queue).

I know why. It's because my code to grab a single message off the queue
looks something like this :

//m_folder is a DirectoryInfo object
FileInfo file = m_folder.GetFiles("*.xml")[0];

So, it's pulling info about every file in the queue _every time_ it
wants to get a message from the queue.

Where I'm struggling, though, is finding a more efficient way to get the
first file in the directory.

One approach I can think of is to do some in-memory caching of the list
of files, so that I don't read them from disk every time. However,
there's a fair amount of effort involved there, because my program
reading the messages from the queue is multithreaded.

So, before I go down that route I thought I'd ask around. Anyone know
of a very efficient way to just pull a single file from a directory?

Thanks,
Craig
 
Sounds like what you need is a variation of the dir command switches.

From the command line, you can do:

dir /od

To sort by the data timestamp and then you could pick off just the first
file. Maybe DirectoryInfo will let you sort by date timestamp.


Craig said:
Hi All,

I have a program that's using the file system as a queuing mechanism,
and it's consuming an inordinate amount of CPU time when the file system
queue gets all that large (any more than a thousand or so messages in
the queue).

I know why. It's because my code to grab a single message off the queue
looks something like this :

//m_folder is a DirectoryInfo object
FileInfo file = m_folder.GetFiles("*.xml")[0];

So, it's pulling info about every file in the queue _every time_ it
wants to get a message from the queue.

Where I'm struggling, though, is finding a more efficient way to get the
first file in the directory.

One approach I can think of is to do some in-memory caching of the list
of files, so that I don't read them from disk every time. However,
there's a fair amount of effort involved there, because my program
reading the messages from the queue is multithreaded.

So, before I go down that route I thought I'd ask around. Anyone know
of a very efficient way to just pull a single file from a directory?

Thanks,
Craig
Click to expand...
 
John said:
Sounds like what you need is a variation of the dir command switches.

From the command line, you can do:
Click to expand...

You could use this:

http://www.shanebauer.com/Weblog/PermaLink,guid,b9cd286d-e147-4a25-aa49-bde888a11433.aspx

I'm assuming you are doing a FIFO queue so getting the earliest date
would work for you.
dir /od

To sort by the data timestamp and then you could pick off just the first
file. Maybe DirectoryInfo will let you sort by date timestamp.


Craig said:
Hi All,

I have a program that's using the file system as a queuing mechanism,
and it's consuming an inordinate amount of CPU time when the file
system queue gets all that large (any more than a thousand or so
messages in the queue).

I know why. It's because my code to grab a single message off the
queue looks something like this :

//m_folder is a DirectoryInfo object
FileInfo file = m_folder.GetFiles("*.xml")[0];

So, it's pulling info about every file in the queue _every time_ it
wants to get a message from the queue.

Where I'm struggling, though, is finding a more efficient way to get
the first file in the directory.

One approach I can think of is to do some in-memory caching of the
list of files, so that I don't read them from disk every time.
However, there's a fair amount of effort involved there, because my
program reading the messages from the queue is multithreaded.

So, before I go down that route I thought I'd ask around. Anyone know
of a very efficient way to just pull a single file from a directory?

Thanks,
Craig
Click to expand...
Click to expand...
 
Hi Craig,

have you considered having a service that uses FileSystemWatcher to maintain
a listing of the files in your directory?
The list could be as simple as a text file in that directory through to any
level of sophistication you might like to best fit your overall applciation
architecture.
It would at least allow you to go directly to a single point for your next
file.

Good Luck
Gerard
 
I would be inclined to use a combination of a FileSystemWatcher component
and a Queue object.

At application startup, start the FileSystemWatcher to wtach for new files
in your target folder, then use the GetFiles method and enqueue each
filename to the Queue object.

When your application wants to get a message from the queue simply

- Check the length of the Queue object (the number of entries)

- Dequeue the first entry into a string (filename)

- Check for the existence of the file

- Process it if it exists

The check for the existence of the file will cater for the potential for the
same filename to get put in the Queue object twice if the FileSystemWatcher
happens to fire bewteen the time you start it and the time you execute the
GetFiles method.

In my view it is best to do it that way round because if you do it the other
way round the is the potential for a file to be missed of it is created
between executing the GetFiles method and starting the FileSystemWatcher.
 
Back
Top