How To Monitor A File For Changes

  • Thread starter Thread starter MarcJ
  • Start date Start date
M

MarcJ

Hello,

What I'm trying to do is monitor a log file for a certain line of text
to be written to it. Once that text is written, I want to parse from
that instance of the watched text until it reaches another line of
specified text. I know I can use the FileSystemWatcher class to see
when the log file is changed, I just can't figure out how to retrieve
what text was appended to the log file since the previous change. This
log file is *very* active and would basically have text written to it
constantly. I need to trigger my parser on every occurrence of the
watched line of text being written to the log file, even if it's written
hundreds of times a day.

Anyone have any suggestions or links on how I might go about doing this?

Thanks!
 
MarcJ said:
What I'm trying to do is monitor a log file for a certain line of text
to be written to it. Once that text is written, I want to parse from
that instance of the watched text until it reaches another line of
specified text. I know I can use the FileSystemWatcher class to see
when the log file is changed, I just can't figure out how to retrieve
what text was appended to the log file since the previous change. This
log file is *very* active and would basically have text written to it
constantly. I need to trigger my parser on every occurrence of the
watched line of text being written to the log file, even if it's written
hundreds of times a day.
Click to expand...

Why not read the contents of the file every time you get a notification
that the content changed and compare it to the previous version of the file?
 
hirf-spam-me- said:
Why not read the contents of the file every time you get a notification
that the content changed and compare it to the previous version of the file?
Click to expand...

This log file would probably be around 200MB. In a real-time situation,
I don't think I can use that method without a HUGE hit to performance
considering the file is being written to several times a second.
 
MarcJ said:
This log file would probably be around 200MB. In a real-time situation,
I don't think I can use that method without a HUGE hit to performance
considering the file is being written to several times a second.
Click to expand...

The stream class provides a 'Seek' method. You can save the length
before the manipulation, then open the file, seek to the position and
read only the text after this position.
 
Hi Marc,

I wonder what 'very active' and 'constantly' are. Assuming that hundreds
of times a day is, say, 3600, max, that's only once every 24 seconds.

At that sort of frequency, it seems reasonable to open the file, read the
set of last lines that were added and give them the once over.

Method.
Start of program.
Note the length of the log file.
Kick off your FileWatcher.
Do something amazing to get the User gasping.

FileWatcher.OnHey!!Look!!TheLog'sChanged!!
Open the log file.
Seek to the previous length.
Read to the end.
Note the new length of the log file.
Examine the text at your leisure.

For text files it is usual to use a StreamReader but this doesn't allow
random access. You could simply read the file line by line until you reach the
desired position. How big do these logs get? ;-)

To do it properly, you need a FileStream and its Seek method. FileStream
reads bytes rather than text so you'll then hook up with an Encoding which can
convert your array of bytes into a String. Choose the Encoding depending on
what character set the log file is written in (probably Ascii, maybe Unicode).

Something like the code below, in fact. ;-)

Regards,
Fergus

<code>
'Note that iPosition is a ByRef parameter.
Private Function SeekAndYeShallFind (sFilePath As String, _
ByRef iPosition As Integer) As String

Dim oFileStream As New FileStream (sFilePath, FileMode.Open)
Dim iFileLength As Integer = CInt (oFileStream.Length)
Dim iNumBytes As Integer = iFileLength - iPosition
If iNumBytes = 0 Then _
Return "" 'Nothing to report, Sah!

Dim aBytes (iNumBytes) As Byte
oFileStream.Seek (iPosition, SeekOrigin.Begin)
oFileStream.Read (aBytes, 0, iNumBytes)

'Return the new file position and the new text.
iPosition = iFileLength
Return Encoding.ASCII.GetString (aBytes)
End Function
</code>
 
Hi again Marc,

Ahah! it's not hundreds - it's hundreds of thousands!!

Lol, you most definititely <don't> want a sequential StreamReader!!

Do you think the FileStream would keep up?? Is there a chance that the
file could be locked while you are attempting to read? You'll have to wrap
that code up in some error-handling, re-try stuff.

Regards,
Fergus
 
Hi Marc, yet again,

I'm wondering whether your text section will be written in one go or will
it span two or more writes?

If the former, then String.IndexOf will suffice. If the latter then you'll
have to cache the text.

It's quite an interesting challenge you've got here. ;-)

Regards,
Fergus
 
Hi Fergus,
Are you not afraid with this code that the log file get locked.
I was thinking on first copying the logfile and then access, but I am not
sure if that is necessary.
I hope to see you tomorrow again.
Cor
 
Hi Fergus,
I saw you discovered it yourself, so see this as not written.
Except of course the last row.
Cor
 
Hi Herfried,

100s x 1000s of bytes in total but not per update I would hope!

And 100s x 1000s of times per day (at several times a second).

Regards,
Fergus
 
Hi MarcJ,

If your log file is manipuate frequently, may be you need use some Reading
Writing synchronization to protect your Watcher process.
You can refer to the file lock flag.

To get the append text, I think you should compare the original file with
the new file.
But if your file is too big, may be the suitable way is use certain File
filter driver.

For more information of File filter driver, I think you should consult in
the SDK or DDK group.

Hope this helps,

Best regards,
Jeffrey Tan
Microsoft Online Partner Support
Get Secure! - www.microsoft.com/security
This posting is provided "as is" with no warranties and confers no rights.

--------------------
| From: MarcJ <[email protected]>
| Subject: How To Monitor A File For Changes
| Date: Wed, 8 Oct 2003 14:09:35 -0400
| Message-ID: <[email protected]>
| Organization: CertifiChecks, Inc.
| X-Newsreader: MicroPlanet Gravity v2.30
| Newsgroups: microsoft.public.dotnet.languages.vb
| NNTP-Posting-Host: rrcs-central-24-106-102-45.biz.rr.com 24.106.102.45
| Lines: 1
| Path: cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP11.phx.gbl
| Xref: cpmsftngxa06.phx.gbl microsoft.public.dotnet.languages.vb:145021
| X-Tomcat-NG: microsoft.public.dotnet.languages.vb
|
| Hello,
|
| What I'm trying to do is monitor a log file for a certain line of text
| to be written to it. Once that text is written, I want to parse from
| that instance of the watched text until it reaches another line of
| specified text. I know I can use the FileSystemWatcher class to see
| when the log file is changed, I just can't figure out how to retrieve
| what text was appended to the log file since the previous change. This
| log file is *very* active and would basically have text written to it
| constantly. I need to trigger my parser on every occurrence of the
| watched line of text being written to the log file, even if it's written
| hundreds of times a day.
|
| Anyone have any suggestions or links on how I might go about doing this?
|
| Thanks!
|
 
Hi again Marc,

Ahah! it's not hundreds - it's hundreds of thousands!!

Lol, you most definititely <don't> want a sequential StreamReader!!

Do you think the FileStream would keep up?? Is there a chance that the
file could be locked while you are attempting to read? You'll have to wrap
that code up in some error-handling, re-try stuff.
Click to expand...

Wow, so many replies! Where to begin...

There is no chance the file will become locked while I'm attempting to
read it.

I would estimate that 6 lines of text, roughly 30 characters on average,
is being constantly written to the log every 1 second.

I'll get back to you guys once I play with some code. I found a program
that does exactly what I'm trying to do, except its written for
something totally different so doesn't support my needs. They call it a
"real-time log file parser".

Thanks!

Marc
 
Hi Marc,

I've decided to give a title to that bit of code that I posted for you**.

It does exactly what you've asked for. It was written for something
totally yours so it should (hopefully) support your needs. I'm going to call
it a "Real-time Log File Reader". You get to parse the 6 lines yourself.

LOL. SCNR. ;-))

Regards,
Fergus

** 8th Oct, 8:29 pm
 
Back
Top