I
K
kurt wismer
Julian said:
ok, it is an anti-virus that lacks most of the sophisticated scanning
technologies present in commercial anti-virus products and therefore
lacks the ability to handle the viruses those sophisticated
technologies were designed to cope with...
J
Julian
kurt said:
That's one way of putting it, but it's a pretty negative way. I don't
see the need to go out of one's way to deride something that a bunch of
public spirited people have put a lot of time and effort into developing
so that they can give it away free.
It seems to do a good enough job of detecting the viruses that are
around today, enough so that lots of people use it to scan the emails
that arrive on their servers. So it isn't all bad.
A
Art
Public spirited people point out the shortcomings of products. Nobody
is going out of their way to deride clamav that I can see. We go out
of our way to help others who haven't tested the products. Period.
Who said it was?
http://home.epix.net/~artnpeg
K
kurt wismer
Julian said:
clamav is an inferior product... there's no way to sugar coat that...
their intentions are not at issue, the ability of the product to detect
viruses is...
good will does not make up for the lack of ability to come up with a
technically competitive package...
if anyone's going out of their way it's you going out of your way to
make excuses for their lack of ability...
only because the current crop of replicative malware is relatively
unsophisticated, and that can quickly change... it hasn't always been
so unsophisticated and it probably won't stay that way...
just because a number of people use it doesn't mean it's any good... a
lot of people used msav, too...
R
Roger Wilco
N
Nick FitzGerald
Julian said:
Why?
The engine is shite and requires significantly more deeply informed
technical input that the open-source community is likely to ever be
able to put into the "product". It is a tooy with some amusing
foibles, which iincludes the ability to detect some viruses and
great scads of crap.
"complete" must have taken on a new menaing while I slept...
How will they keep up with the thousand-plus a month workload for the
dull, dreary, non-sexy virus analysis side of maintaining a "complete"
scanning-based AV engine?
Your results are rubbish and if you have any integrity you'll do two
things -- remove that page and any copies of it and replace it with a
detailed explanation of why anyone who already read it should disregard
the results and any opinions formed as a result of reading them.
You might also consider learning something (anything?) about how
marginally adequate virus detection rate scanning tests are designed
and performed before considering you may be capable of performing
such again in future.
A few things you clearly don't understand...
# False alarms: ClamAV seems more prone to giving false alarms. On
my test sample of 2118 COM and EXE files it produced two false alarms.
It also flagged as "Suspected.Zip" a harmless small Zip file on my
hard drive.
What were those "2118 COM and EXE files"?? One must presume they were
sourced from known good locations -- you should explain the methodology
of sourcing and choosing these files.
2118 files is an appalling small number for FP testing. Sadly for
ClamAV though, it was enough for it to find two clean files "infected".
That kind of FP rate would not kill a general purpose commercial scanner
-- it would see it still born, with no-one ever moving from evaluation
to considering purchase and production use. The best commercial AVs do
internal FP testing against multi-hundred Gigabyte clean file sample
sets as part of their QA -- ClamAV's equivalent of this is???
# Worm detection: ClamAV did better than F-Prot in detecting some of
my collection of worms and other email exploits. It picked up a couple
of (admittedly, old - Kak and Mawanella) worms that F-Prot missed, ...
I seriously doubt that result. Clam's brain-dead grunt scanning almost
certainly is "detecting" some "dead code" or a fragmentary copy, either
of which cannot work (i.e. is non-replicative). Now, whether you would
like your scanner to detect such non-working, non-infecting, unable to
spread by itself code or not, is up to you, but including such broken,
non-working samples in a "worm detection test" is inexcusably sloppy and
shows an enormous ignorance of what you are trying to do and achieve.
... and
was also more effective at detecting GFI's test mail exploits. This is
perhaps not surprising as most of the current users of ClamAV seem to
use it as a scanning service for mail servers. Because of this, I will
be considering changing to using ClamAV to perform virus scanning on
our own home/small office mail server product Tech-Pro Mail Server.
Does F-PROT claim to detect these exploit attempts?
And does ClamAV actually detect anything apart from the precise form of
them as sent out in the GFI test Email messages?
You are trying to answer bigger questions here than "Who detects GFI's
Email test messages?", so please, put a bit of effort into devising a
test-set whose detection results will address the actual question.
# In the wild viruses: ...
Do you have access to the WildList's reference collection so you can be
sure you have samples of the right things? In case you don't know,
some products call what is really, say Bagle.AU, Bagle.AZ, others call it
Bagle.AY and yet others Bagle.BK (Oh, and the WildList does not always
use the "right" name here either). So, how can you tell that something
you have a sample of that some scanner calls by a name that appears in
the WildList is actually a sample of the same thing as is WildList'ed?
As WildList reports are sample-backed, the WildList folk compiling the
lists make the determinations as to what has been reported multiple times,
so access to their reference sample set is necessary for testers to be
sure they are correctly classifying things as WildList'ed or not.
... ClamAV did about as well as F-Prot in scanning
my (admittedly several years old) collection of "in the wild" COM and
EXE viruses. ...
And I thought the whole point of WildList testing was to determine the
currency of detection of "important" malware...
... It picked up one virus that F-Prot missed, ...
And, as in the Kak and Mawanella (BTW, the correct name of that virus is
VBS/VBSWG.Z), I have to ask "And you know that this missed file was a
viable sample of a virus how?"
... but failed to
identify a couple of others. However, it was noticeably worse than
F-Prot at detecting macro viruses, detecting only 8 of my 16 samples.
ClamAV is _utterly_ useless at detcting macro viruses as the Clam engine
developers have no idea what macro viruses look like, how to handle the
fiddly internal (and undocumented) file format structures in which the
macro code resides, and so on.
# Other viruses: ClamAV also did an excellent job of identifying other
COM and EXE viruses that were not on the WildList at the time (a few
years ago) that I classified this virus collection. ...
What was "this virus collection"? You see, to be a useful sample set
for conducting a fair virus detection test, your "virus collection" has
to meet some rather strict criteria. It must contain viruses -- that is,
functionaing self-replicating programs and not just a pile of files that
some or other person or some version of some virus scanenr once said were
viruses. To be sure of this, you (the tester) should have proven to your
own satisfaction that all the files in your test are viruses. This is a
very tall technical requirement that few are capable of attempting...
... It detected more
of these viruses than the F-Prot DOS scanner. ...
And you _know_ these are viruses because???
... However, it was once
again very poor at spotting macro viruses, detecting only 6 of my 36
miscellaneous DOC, DOT, XLS and WK1 samples.
And you know these are viruses because??? (Well, if F-PROT detected them
there is a very high probability they are, as the macro specialist at FSI
is _very_ particular about what he lets the macro detection part of their
engine detect, but even that does not _prove_ they are viruses...)
Your test is a mixture of a crud detection test and a virus detection
test, but as the results on the crud and the virus files are not
differentiated, we cannot draw many useful conclusions about the relative
merits of the products tested.
N
Nick FitzGerald
Julian said:
Aside from actually cutting code (which the folk you are talking about
get paid to do now and probably mostly have non-compete clauses in their
employment contracts and the like), what could have been more helpful
than this:
http://www.securityfocus.com/infocus/1650
Costin is probably the last person who will ever write a "complete" virus
detection engine from scratch and is among a smallish group (fewer than
20 folk I'd guess) who has _ever_ done that for a modern-ish AV engine.
Costin started more or less from scratch when most of the existing engines
were being broken into smaller component parts of re-designed scanning
architectures and most of those who had written the engines that came
before his had already become much more specialist in their focus, or
retired. He has a very good appreciation of what makes a good engine,
what doesn't and where OAV (and hence Clam) has to go to get there from
where they were when he wrote that article. They've nearly entirely
ignored his advice...
Last time there was a "completely free" AV that came on all new (DOS and
DOS/Windows) machines, most new computer buyers, who never read magazines
or visit technical fora, had to get a decent AV product *after* the first
time they became infected. MS probably cannot afford to try to make that
mistake again (including a free AV with the OS) as the anti-competition
law suits will be fought against companies that have much deeper pockets
than Netscape _and_ substantial corporate loyalty on their side...
I seriously doubt we will ever see that happen.
N
Nick FitzGerald
kurt wismer said:
But much worse is that, once all the low-hanging fruit has been picked by
today's crop of simplistically naive malware, it is just those sites that
have settled on AV products with totally inadequate technological bases,
such as Clam-based Email gateway scanners, that will be blitzed by the
next wave of "more complex" malware.
_THAT_ is why setting up folk to use grivously inadequate technology
that seems to work today is so bad. Doing so actively cultivates the
next field for attack...
And how many billions use IE as their web browser?
K
kurt wismer
Nick said:
in other words, clam av gives people a false sense of security and
eventually they'll wind up facing the consequences...
that's generally where the 'good _enough_ for today's threats' argument
leads, i think... good enough for the here and now basically means
'stop-gap measure' and those can't be relied on for the long term...
J
Julian
Nick,
Thank you for your input. I think, actually, the crap has been pretty
much filtered out of my collection having been scanned in the past by
many different scanners during the course of my doing reviews for
magazines. Anything that didn't get reliably detected by respected
products like F-Prot, Sophos etc. got tossed out long ago.
I accept the fact that any test I carried out in a few hours one
afternoon without access to better virus test suites may have flaws,
but I still feel that it has some value as an attempt to quantitatively
assess the performance of the ClamAV scanner because as far as I know,
no-one with greater authority has published tests on it. Therefore, the
only indication of its abilities are aniecdotal comments from people
like yourself who, for all I know, may have some hidden aganda or axe to
grind over it.
The internet is full of people happy to criticize the work of others
without ever producing anything of value themselves. At least I made an
honest attempt to evaluate the performance of this scanner objectively
whereas you simply make statements like "the engine is shite" and "It is
a tooy with some amusing foibles" with no supporting evidence whatsoever.
Thank you for your input. I think, actually, the crap has been pretty
much filtered out of my collection having been scanned in the past by
many different scanners during the course of my doing reviews for
magazines. Anything that didn't get reliably detected by respected
products like F-Prot, Sophos etc. got tossed out long ago.
I accept the fact that any test I carried out in a few hours one
afternoon without access to better virus test suites may have flaws,
but I still feel that it has some value as an attempt to quantitatively
assess the performance of the ClamAV scanner because as far as I know,
no-one with greater authority has published tests on it. Therefore, the
only indication of its abilities are aniecdotal comments from people
like yourself who, for all I know, may have some hidden aganda or axe to
grind over it.
The internet is full of people happy to criticize the work of others
without ever producing anything of value themselves. At least I made an
honest attempt to evaluate the performance of this scanner objectively
whereas you simply make statements like "the engine is shite" and "It is
a tooy with some amusing foibles" with no supporting evidence whatsoever.
J
Julian
Thanks for your comments and the link, Nick.
T
Tomasz Kojm
Dear Sir,
please point your browser to www.clamav.net, download the latest stable
release, read the source code, and stop defaming us.
I'm not going to reform you, personally I don't care what is your opinion
on open-source software and community. However I was meeting your rude
comments since 2002, the same things, over and over again. The worst thing
is they're not only rude but they're simply lies.
You're declaring yourself as an independent consultant. Being a professional,
you should provide an objective and truthful information. Next time you
are going to express your opinion on ClamAV, please first verify your
knowledge to avoid statements that are slanders and nothing more.
Tomasz Kojm
please point your browser to www.clamav.net, download the latest stable
release, read the source code, and stop defaming us.
I'm not going to reform you, personally I don't care what is your opinion
on open-source software and community. However I was meeting your rude
comments since 2002, the same things, over and over again. The worst thing
is they're not only rude but they're simply lies.
You're declaring yourself as an independent consultant. Being a professional,
you should provide an objective and truthful information. Next time you
are going to express your opinion on ClamAV, please first verify your
knowledge to avoid statements that are slanders and nothing more.
Tomasz Kojm
K
kurt wismer
Julian said:
which basically tells us you have no clue... scanners are great crap
detectors precisely because magazine reviews use crap in their
detection tests and it's the only way those scanners can hope to be
viewed positively in that context...
try holes big enough for planetary bodies to pass through...
and for all i know, nick may have actually been part of an independent
evaluation of the product during his stint as an editor for virus
bulletin... it certainly wouldn't surprise me if he had...
never mind the fact that clamav boasts detection of a number of viruses
so small as to make it hardly worth the effort to test it...
yes well... i imagine when one is an authority on a subject, one tends
to get out of the habit of providing supporting documentation... of
course the fact that one probably wrote at least some of the supporting
documentation sort of dulls the point anyways...
J
Julian
kurt said:
You're wrong there. Indeed, I can recall several long conversations with
AV company personnel who were concerned that I was not using "crap"
non-viruses in my tests because their scanners wouldn't detect them.
One time, I was interested to know why a sample hadn't been detected,
because a highly respected scanner had failed to detect one of my ITW
virus samples. I sent it to their laboratory, they analyzed it, and said
that the reason it wasn't detected is because it was not a genuine
replicant. The sample had something like 15 null bytes on the end,
possibly as a result of being downloaded using some protocol like xmodem.
So in their view, my sample was "crap". I beg to differ. The ability of
this sample to replicate was in no way affected by these 15 null bytes
on the end. *My* definition of a virus is a program that will replicate.
If someone can download an infected file and have it pass by a virus
check simply because the download protocol they used doesn't set the
file size correctly, then in my view that is a failure of the virus
scanner. You're entitled to disagree, but please don't insult me my
saying I "have no clue."
Anyway, that's all history now. These days magazines don't ask me to do
AV reviews. They get youngsters fresh out of media studies courses with
no technical knowledge at all to do 500 word reviews of AV products
based on no testing whatever, so the one that gets recommended is the
one that has the prettiest interface and the most bells and whistles.
R
Roger Wilco
That sample shouldn't have been in a test. You should have treated it as
a seed and used a true and proven to be capable of creating viable
offspring replicant for testing. The virus that the seed file represents
may be an ITW virus, but your sample was not what would be expected ITW.
Replicate it into some other program and use that program as the test
sample only after assuring that it itself is capable of creating viable
(reproducing) offspring
Your definition falls short of the mark, besides - when that sample was
replicated, did the offspring also have the null bytes?
The suspected protocol would probably add the bytes to the host program
file, not to the virus body.
I bet you have a personal different definition for 'infected' too.
Actually, 'infected by a virus' is not the same as 'contains a virus'.
Scanning for viruses usually means looking for infections as they would
occur in the wild, not looking for viral code contained in a downloaded
file (which is more like a worm or trojan detection thing).
J
Julian
Roger said:
No. The bytes were added to the end of the original file by the
communications protocol, because it didn't know the exact file length.
It's a bit academic, as I doubt anyone uses the xmodem protocol now, but
I don't agree that it was not a valid sample. Suppose I'm just an
ordinary guy who downloaded an infected file from a BBS using xmodem: I
would expect my AV to detect it right away, not after I'd run it and it
had infected something else and created some genuine replicants.
Consider: since the AV isn't detecting the original file that caused the
infection, I could have the virus keep coming back unless I figure out
where it must have come from.
K
kurt wismer
Julian said:
no, because their scanners *might* not detect them... they obviously
have a lot more control over whether their scanners can detect viruses
than they do over whether their scanners can detect some arbitrary
piece of crud used in a brain-dead magazine review... so yes, if you
were doing a high (enough) profile review, you're going to get
contacted by folks who would prefer that you not use crud...
that doesn't change the fact that scanners have been forced to detect
crud by brain-dead magazine reviewers who use crud in their tests and
therefore scanners cannot be used to weed out crud...
lets look at this a little more analytically, shall we? it wasn't
detected because it had been modified... at best it was a new variant
(did they add detection?) but more likely it was damaged... in any
case, modifications have a habit of making a virus not match it's
signature anymore...
you say the null bytes didn't affect it's ability to replicate - do you
know this for a fact? were you able to get 2nd generation offspring
from it?
if they said "it's not a virus anymore" and you got 2nd generation
offspring from it then it's your duty to name and shame, otherwise
you're blowing smoke...
you claim to have weeded the crud out of your collection by using
scanners - there's little room for any other interpretation... the only
legitimate way to weed crud out of a collection is to replicate the
samples to at least the 2nd generation or, if you happen to be an
expert virus analyst, you may be able to tell simply by looking at the
code (however, 2nd gen. offspring or lack thereof trump that)...
which, more and more, is a perfectly reasonable thing for magazines to
do... they don't have the resources to produce detection tests and even
if they did, the major players are close enough to each other in terms
of raw detection rates that those other properties are now much more
legitimate criteria for deciding which av to use...
R
Roger Wilco
Julian said:
Some viruses, in the course of their 'lives', have a multitude of
different forms they permutate through from iteration to iteration. The
AV scanning engine should detect those forms and not necessarily the
other less likely natural mutations it might encounter. Besides, you
said yourself that the nulls weren't passed on the the progeny, so it
(your sample) wasn't a known form of that virus. If the virus had been
able to pass the nulls on to its progeny, it would have to be considered
a 'new' strain of that virus.
I see what you mean, however an 'infected' file once modified (after
infection has taken place) - and modified so that an AV doesn't detect
it - is more trojan (accidental trojan?) than virus since it is not the
usual form.
Academic as you say - especially since you didn't name the virus.
The original 'seed' or 'dropper' can vary far outside the realm of where
the virus would be expected to reside. A virus that infects only exe
files can itself be contained in a dropper that is a com file. If the
dropper is common (like a worm) then the AV probably would detect it -
but the less common random natural mutation would go unnoticed until it
itself got recognised as a variant.
Most any programmer could alter a virally infected file so that it
sneaks past the AV - and there are probably many more of those then
there are of xmodem accidental trojans. And since neither of these
modifications turn up in the offspring, why should AV detect them? Your
sample of xmodem modified virus might be the only one in existence, and
you expect it to be detected? IMO that is not very realistic.
N
Nick FitzGerald
"kurt wismer" <[email protected]> to Julian:
This is sadly true, even for some of the most respected scanners...
Without knowing the specifics, it is _very_ hard to tell on a case by case
basis. I have several binary different files that are all "proper" samples
of Netsky.P and the same with Swen.A. I received these through normal,
virus-generated Email messages addressed to my normal Email address. Most
of these are trivial variations on the "base" Netsky.P and Swen.A samples
-- for those not in the know, these viruses _should_ produce binary
identical replicants as they simply take a copy of themselves, encode it
and send it as an Email attachment. The "odd" samples I have (mostly) have
one (or more) null, CR, LF characters or CRLF pairs appended (it seems a
fair bet that most of these have been created at some point or other as
Email copies of "proper" samples of these viruses have passed through some
dodgy (home-grown?) gateway scanners whose unpackaging/decoding software
does not properly deal with some "odd" boundary conditions of the decoding
algorithm, but this is unproven -- MessageLabs sees many, many more minor
variations such as this with each outbreak). All scanners I've tested have
detected these "variant samples" (but note that these are NOT considered to
be new variants!).
Simlarly, back in the days of script viruses, we saw truncated but still
functional viruses (for example, the first mIRC script virus could, from
memory, be truncated anywhere from line 52 (??) onwards it would remain
replicative -- some scanners detected up to that point, others classified
all different "chopped" samples as new variants) and mass-mailers that
unpacked in various partially munged ways (or with trailing additional
line breaks, EOF chars, nulls, etc) but which also remained perfectly
functional (thanks to VB's "on error resume next"...).
I can imagine (and have seen) many instances where this could be the case.
Further, I can imagine cases where the result would be samples that had
"reverted to form" and cases where the same, or yet another, binary
different (though not necessary different variant) form would result.
You can understand a degree of animosity to entirely artifically created
samples of this nature, but if the forms of interest here really are
"naturally occurring", or at least, fairly likely to occur as the result
of "normal use", then the scanner vendors should be aware of both the
possibility _and_ have coded around that in their detection engines. As
just one example of a case where this has been done, after the second
generation versions of some MS Office Suite applications that supported
VBA macros were released, the AV industry discovered that certain new VBA
features that were, of course, non-supported in the earlier versions of
VBA, caused additional "blank line" operators to be added to the head of
the VBA code generated in the earlier VBA versions. Thus, when an Excel
97 (VBA 5.0), say, spreadsheet was opened in Excel 95 (VBA 3.0 ?? VBA <
VBA 5.0 anyway), the latter "inserted" an additional blank line at the
head of the VBA source in its representation of the macros in the
spreadsheet then compiled the binary forms it would actually execute. If
the Excel 95 form was then taken back to Excel 97, Excel 97 would retain
that "extra" blank line. Several engine's VBA parsers had to be quickly
fixed to ignore _leading_ blank lines (don't ask why lines were not
necessarily ignored to start with...).
This is sadly true, even for some of the most respected scanners...
Without knowing the specifics, it is _very_ hard to tell on a case by case
basis. I have several binary different files that are all "proper" samples
of Netsky.P and the same with Swen.A. I received these through normal,
virus-generated Email messages addressed to my normal Email address. Most
of these are trivial variations on the "base" Netsky.P and Swen.A samples
-- for those not in the know, these viruses _should_ produce binary
identical replicants as they simply take a copy of themselves, encode it
and send it as an Email attachment. The "odd" samples I have (mostly) have
one (or more) null, CR, LF characters or CRLF pairs appended (it seems a
fair bet that most of these have been created at some point or other as
Email copies of "proper" samples of these viruses have passed through some
dodgy (home-grown?) gateway scanners whose unpackaging/decoding software
does not properly deal with some "odd" boundary conditions of the decoding
algorithm, but this is unproven -- MessageLabs sees many, many more minor
variations such as this with each outbreak). All scanners I've tested have
detected these "variant samples" (but note that these are NOT considered to
be new variants!).
Simlarly, back in the days of script viruses, we saw truncated but still
functional viruses (for example, the first mIRC script virus could, from
memory, be truncated anywhere from line 52 (??) onwards it would remain
replicative -- some scanners detected up to that point, others classified
all different "chopped" samples as new variants) and mass-mailers that
unpacked in various partially munged ways (or with trailing additional
line breaks, EOF chars, nulls, etc) but which also remained perfectly
functional (thanks to VB's "on error resume next"...).
I can imagine (and have seen) many instances where this could be the case.
Further, I can imagine cases where the result would be samples that had
"reverted to form" and cases where the same, or yet another, binary
different (though not necessary different variant) form would result.
You can understand a degree of animosity to entirely artifically created
samples of this nature, but if the forms of interest here really are
"naturally occurring", or at least, fairly likely to occur as the result
of "normal use", then the scanner vendors should be aware of both the
possibility _and_ have coded around that in their detection engines. As
just one example of a case where this has been done, after the second
generation versions of some MS Office Suite applications that supported
VBA macros were released, the AV industry discovered that certain new VBA
features that were, of course, non-supported in the earlier versions of
VBA, caused additional "blank line" operators to be added to the head of
the VBA code generated in the earlier VBA versions. Thus, when an Excel
97 (VBA 5.0), say, spreadsheet was opened in Excel 95 (VBA 3.0 ?? VBA <
VBA 5.0 anyway), the latter "inserted" an additional blank line at the
head of the VBA source in its representation of the macros in the
spreadsheet then compiled the binary forms it would actually execute. If
the Excel 95 form was then taken back to Excel 97, Excel 97 would retain
that "extra" blank line. Several engine's VBA parsers had to be quickly
fixed to ignore _leading_ blank lines (don't ask why lines were not
necessarily ignored to start with...).