Using ClamAV as a general purpose scanner

[Roger Wilco]

The tested version was CLAM AntiVirus v:0.54 - the current stable
release is 0.82. Please tell me that you didn´t build your opinion on
such an outdated result.The ability to handle macro-viruses was first
implemented in 0.70. Somehow this is getting more and more ridiculous.

The headline of this test was: AntiVirus Scanner Tests July 2004
The 0.70 was released in march 2004. The 0.65 in november 2003.

But i guess as long as it supports the halftruths, speculations and
presumptions, even such a at least strangely outdated test is welcome.

Old pictures CAN be unflattering.

"detection of over 30000 viruses, worms and trojans" is even now on
http://www.clamav.net/abstract.html#pagestart.

Is this lack of definitions, or lack of technology? Isn't it an accepted
fact that there are in excess of 70,000 viruses and who knows how many
trojans out there. 30,000 from the group virus/worm/trojan may sound
impressive to the average joe, but it makes me think there is something
still wrong with general acceptance as a general purpose AV.

....an old page? 1999-2000? What's the new figure?

 

[Christoph Cordes]

I see nothing strange (other than my earlier typo) about the dates of
the test.

The strange thing is, that the opinions about the current stable release
are based on tests of simply outdated releases. The ClamAV development
is still in progress, there have been a lot of changes, improvements and
bugfixes. So if you want to judge about the performance of the current
release, you have to use a test that is a _bit_ more recent or just try
it yourself. Of course there is nothing strange about your comment, the
only strange thing is the testing methodology - not only Clam - all AV
products are under a continuous development, so - and that´s my personal
opinion - i don´t see any use for a test results that shows the
performance of an AV months ago - or in other words - would you base
your purchase decision on such a test or would you at least trial the
current version to find out if it fits your needs?

So at the moment there are no facts - only presumptions that are based
on very old results.

 

[=?ISO-8859-1?Q?=BBQ=AB?=]

Of course there is nothing strange about your comment, the
only strange thing is the testing methodology - not only Clam -
all AV products are under a continuous development, so - and
thatïs my personal opinion - i donït see any use for a test
results that shows the performance of an AV months ago - or in
other words - would you base your purchase decision on such a test
or would you at least trial the current version to find out if it
fits your needs?

I would indeed base my decision on such a test, or preferably on a
series of such tests over the years. Past performance in such rigorous
tests has been a pretty good indicator of future performance, so I
don't mind extrapolating. Trialing the current version tells me
nothing about its detection rates; the only use of a trial for me is
making sure the interface isn't awful.

If ClamAv really is improving drastically, then that will show up in
further testing, and people will notice. As of now, though, the only
indicator that ClamAv is worth using as a general purpose scanner is a
series of Usenet posts saying so. Not that I don't trust Usenet.

 

[Christoph Cordes]

Old pictures CAN be unflattering.

"detection of over 30000 viruses, worms and trojans" is even now on
http://www.clamav.net/abstract.html#pagestart.

Is this lack of definitions, or lack of technology? Isn't it an accepted
fact that there are in excess of 70,000 viruses and who knows how many
trojans out there. 30,000 from the group virus/worm/trojan may sound
impressive to the average joe, but it makes me think there is something
still wrong with general acceptance as a general purpose AV.

....an old page? 1999-2000? What's the new figure?

Do you realy believe that the number of signatures tells anything about
the performance of an AV. There is an AV with about 118000 signatures
which is considered one of the best - if not the best AV out there -
then there is another AV with about 149.000 signatures - but it can
hardly reach the detection of the first. Still both are very good
programms, but they follow different approaches. And based on the number
of signatures, you can tell the following about the quality of this
products: nothing.

With 30.000 signatures there is no way to compete with them, but it´s by
far enough to do the job it was made for and even a bit more.

OK - what´s next? Maybe we can draw a conclusion by reading the
horoscope of one of the developers?

 

[Frederic Bonroy]

Christoph Cordes a écrit :

Do you realy believe that the number of signatures tells anything about
the performance of an AV. There is an AV with about 118000 signatures
which is considered one of the best - if not the best AV out there -
then there is another AV with about 149.000 signatures - but it can
hardly reach the detection of the first. Still both are very good
programms, but they follow different approaches. And based on the number
of signatures, you can tell the following about the quality of this
products: nothing.

With 30.000 signatures there is no way to compete with them, but it´s by
far enough to do the job it was made for and even a bit more.

Depends on what kind of signature they are, no? Judging by
signatures.pdf, Clamav's signatures are more or less sophisticated hex
search strings. So, can we draw the conclusion that Clamav heavily
relies on pattern matching and that there is no emulation for example?

 

[Art]

Do you realy believe that the number of signatures tells anything about
the performance of an AV.

The point, I believe, is that the CLAM developers aren't claiming
anything else but. It's stated plainly that CLAM detects a little over
30K viruses. Real av products are now claiming way over 100K.

Fortunately, average users (most users) want realtime av. Since Clam
Win doesn't (currently) have a realtime monitor component, it will not
be widely used .... and that's a very good thing. It's also as slow as
molasses scanning on-demand, and that will turn off users as well.
So it seems the internet is safe enough so far from Clam


http://home.epix.net/~artnpeg

 

[Roger Wilco]

Do you realy believe that the number of signatures tells anything about
the performance of an AV.

I didn't say signatures, but if you are saying that the number of
signatures = the number of viruses/worms/trojans detected -yes I do
think so to a point. If an AV stated it is now capable of detecting 120
viruses/worms/and trojans wouldn't you think so too?

There is an AV with about 118000 signatures
which is considered one of the best - if not the best AV out there -
then there is another AV with about 149.000 signatures - but it can
hardly reach the detection of the first.

I'm not ready to accept the number of signatures is the same as the
number of viruses/worms/trojans AV will detect, but the statement I
quoted from that page made no mention of signatures. It assigned a
number to the set of malwares it was capable of detecting (30,000). I
believe there are much more than that in existence. It is easier to
detect a virus than to identify one, and an increase in definitions may
be more for identification and removal than for detection - but still
30,000 seems small to me.

Still both are very good
programms, but they follow different approaches. And based on the number
of signatures, you can tell the following about the quality of this
products: nothing.

I agree (it sounds logical), but that's not the issue. The number of
detectable malwares does bear a loose relationship with the number of
definitions - this is why I asked if it was a lack of definitions
(perhaps due to a lack of samples to analyse and produce definitions
for), of a lack of technology (such as was suggested wrt macrovirus,
polymorphic or polymorphic decryptor detection). I said nothing about
"no. of signatures = no. of detections".

With 30.000 signatures there is no way to compete with them, but it´s by
far enough to do the job it was made for and even a bit more.

That's okay, but is that good enough to be general purpose AV. Its high
detection rate in its specialized environment is largely due to the
plethora of e-mail worms that don't make it particularly difficult for a
more simple scanner to detect.

OK - what´s next? Maybe we can draw a conclusion by reading the
horoscope of one of the developers?

I bet YOU could.

 

[Julian]

Take a look at this:
ftp://agn-www.informatik.uni-hamburg.de/pub/texts/tests/pc-av/2004-07/a2scanls.txt

and then this:
ftp://agn-www.informatik.uni-hamburg.de/pub/texts/tests/pc-av/2004-07/0xecsum.txt

where it reads:
"VTC test '2004-07' was started in June 2003, with tesbeds frozen as
known on December 31, 2003 and products submitted in February 2003 (test
start was delayed as HEUREKA-3 test was completed only in May 2003)."


It looks like the other products were equally outdated (due to time
constraints: you don't do a huge test like this on a rainy sunday
afternoon) and no product got a special treatment.

You're missing the point. Christoph stated that the version of ClamAV
tested did not have the capability to detect macro viruses. This was
incorporated until later. It is not an issue of not being able to detect
newer viruses than the software being tested, as you seem to be suggesting.

 

[Julian]

AV apps' reputations are built on their testing track records, whether
you find it ridiculous or not. ClamAv has a short and very bad one;
perhaps that will change.

The 0.x version number should tell anyone that Clam is still a beta
product. Therefore any shortcomings exposed by testing should be viewed
as things that need to be addressed in future releases, not used forever
as a stick to beat it with.

 

[Julian]

Is this lack of definitions, or lack of technology? Isn't it an accepted
fact that there are in excess of 70,000 viruses and who knows how many
trojans out there. 30,000 from the group virus/worm/trojan may sound
impressive to the average joe, but it makes me think there is something
still wrong with general acceptance as a general purpose AV.

And how many of those viruses are samples that have never been seen in
the wild, including ones that the analyses say things like "can only be
persuaded to replicate with great difficulty"? The only value these
claims of total number of viruses detected have are to the marketing
department, where it helps to bolster the claims of the big-name
products at the expense of newcomers that are not "in the loop" for the
exchange of virus samples. Do the commercial AV developers pass new
virus samples to the Clam / Open AV people, like they allegedly do with
each other?

 

[Frederic Bonroy]

Julian a écrit :

You're missing the point. Christoph stated that the version of ClamAV
tested did not have the capability to detect macro viruses.

How did it manage to detect those 40 macro viruses then (0.5%)?

ftp://agn-www.informatik.uni-hamburg.de/pub/texts/tests/pc-av/2004-07/6llin.txt

 

[Frederic Bonroy]

Julian a écrit :

And how many of those viruses are samples that have never been seen in
the wild,

Take a look at this: http://www.securityfocus.com/infocus/1813

Quote:
"You may rightfully ask: why does it matter to detect such viruses, if
they belong to "zoo" collections? Well, first of all, sometimes they do
find their way into the wild. W32/Toal, for instance, a difficult
polymorphic worm, was discussed on an emergency virus mailing list after
being spotted actively spreading. Some complex viruses currently
registered as zoo samples spread aggressively enough that they would
stand a chance to infect machines in the real world if some mischievous
soul were to release them.

Moreover, even for purely zoo viruses unlikely to ever cause problems in
the wild, the response (or lack thereof) of AV companies to such viruses
can reveal a lot about limitations in the engine technology available,
and perhaps the skill and dedication of the response teams."

 

[Julian]

Julian a écrit :




How did it manage to detect those 40 macro viruses then (0.5%)?

ftp://agn-www.informatik.uni-hamburg.de/pub/texts/tests/pc-av/2004-07/6llin.txt

He actually stated that it didn't have the "ability to *handle*
macro-viruses". I interpreted that (perhaps wrongly) as meaning that it
wasn't able to distinguish a macro from anything else in the file. My
guess is that it could still detect a macro virus using simple pattern
matching.

 

[Guest]

Julian a écrit :


How did it manage to detect those 40 macro viruses then (0.5%)?

There's more than one way to detect macro viruses. The 'best' way would be to
completely understand the file format of Office documents, such that the VBA
source code, p-code and exe code can all be decoded, and checked for viruses.

The most naive way is to not understand the file format at all, and simply
match against parts of binary office files.

ClamAV used to lack understanding of the Office file formats and do naive
matching against binary parts.

Versions since 0.70 pull the OLE2 containters apart and extract the VBA
source code for scanning (and in a few cases match against the p-code).

-tony

 

[Julian]

Take a look at this: http://www.securityfocus.com/infocus/1813

Quote:
"You may rightfully ask: why does it matter to detect such viruses, if
they belong to "zoo" collections? Well, first of all, sometimes they do
find their way into the wild. W32/Toal, for instance, a difficult
polymorphic worm, was discussed on an emergency virus mailing list after
being spotted actively spreading. Some complex viruses currently
registered as zoo samples spread aggressively enough that they would
stand a chance to infect machines in the real world if some mischievous
soul were to release them.

Moreover, even for purely zoo viruses unlikely to ever cause problems in
the wild, the response (or lack thereof) of AV companies to such viruses
can reveal a lot about limitations in the engine technology available,
and perhaps the skill and dedication of the response teams."

Well, the would say that, wouldn't they? The major AV companies are
involved in a marketing-led drive to demonstrate that their products are
better. And it doesn't serve the interests of those who write about this
business to argue that these claims are pointless.

One often reads of people using one of the well-known products, who
still get infected by a virus. So advanced engine technology is no
guarantee of effectiveness. Arguably, the most practically effective AV
in today's conditions is the one that is fastest in responding to new
viruses.

 

[Frederic Bonroy]

Julian a écrit :

One often reads of people using one of the well-known products, who
still get infected by a virus. So advanced engine technology is no
guarantee of effectiveness. Arguably, the most practically effective AV
in today's conditions is the one that is fastest in responding to new
viruses.

The issue is that if your engine isn't sufficiently sophisticated to
deal with complex threats, then one day you're going to be in real
trouble when you're going to have to respond quickly to a virus that
goes far beyond what your engine is capable of handling.

 

[Julian]

The issue is that if your engine isn't sufficiently sophisticated to
deal with complex threats, then one day you're going to be in real
trouble when you're going to have to respond quickly to a virus that
goes far beyond what your engine is capable of handling.

No AV developer can see into the future. Even the sophisticated products
often seem to need engine updates. And given that virus writers are
generally regarded as being not very clever, I'm a tad cynical about how
much sophistication is really needed. Virus scanner writing may be
outside of most people's programming experience, but rocket science - I
doubt it.

I think detection of "zoo" virus samples and the sophistication of the
software are two separate matters. Inability to detect samples that
aren't found in the wild may have more to do with not spending time
generating signatures for things that aren't any current threat to
users, or not wanting to waste memory and CPU time scanning for
something you're unlikely ever to find, something that strikes a chord
in those of us who feel that modern AV products are becoming bloated
resource hogs. If one of these "zoo" viruses did become prevalent in the
wild, they would be treated like a new virus and added to the database
quickly.

 

[Frederic Bonroy]

Julian a écrit :

No AV developer can see into the future. Even the sophisticated products
often seem to need engine updates.

Yep - so considering that even highly sophisticated products
occasionally need engine updates, the future doesn't look bright for
Clamav, does it?

And given that virus writers are
generally regarded as being not very clever, I'm a tad cynical about how
much sophistication is really needed.

Exceptions confirm the rule:
http://www.viruslist.com/en/viruses/encyclopedia?virusid=22574

"The virus itself is about 30Kb in length and is written in Assembler,
being very large for a virus written in pure Assembler language. This
large size however is caused by the virus' Win32 EXE files infection
algorithm, e-mail and network spreading routines, polymorphic engines
(there are two), payload routines and many anti-debugging and other
tricks used by the virus to make its detection and disinfection more
difficult. Thus, this virus is one of the most complex viruses that are
known at the moment.

The virus was found in-the-wild in the middle of March 2001."


And you are telling me engines don't have to be particularly
sophisticated? I'm curious to know how reliably Clamav detects this
beast (its .B variant required Frisk to update the F-Prot engine).

I think detection of "zoo" virus samples and the sophistication of the
software are two separate matters. Inability to detect samples that
aren't found in the wild may have more to do with not spending time
generating signatures for things that aren't any current threat to
users, or not wanting to waste memory and CPU time scanning for
something you're unlikely ever to find, something that strikes a chord
in those of us who feel that modern AV products are becoming bloated
resource hogs.

AV bloat is primarily due to HTML interfaces and other similarly idiotic
"innovations".

 

[Julian]

Yep - so considering that even highly sophisticated products
occasionally need engine updates, the future doesn't look bright for
Clamav, does it?

You're making an assumption that it is not going to improve beyond its
current level.

Exceptions confirm the rule:
http://www.viruslist.com/en/viruses/encyclopedia?virusid=22574

"The virus itself is about 30Kb in length and is written in Assembler,
being very large for a virus written in pure Assembler language. This
large size however is caused by the virus' Win32 EXE files infection
algorithm, e-mail and network spreading routines, polymorphic engines
(there are two), payload routines and many anti-debugging and other
tricks used by the virus to make its detection and disinfection more
difficult. Thus, this virus is one of the most complex viruses that are
known at the moment.

The virus was found in-the-wild in the middle of March 2001."


And you are telling me engines don't have to be particularly
sophisticated? I'm curious to know how reliably Clamav detects this
beast (its .B variant required Frisk to update the F-Prot engine).

I don't have a sample, so I can't tell you. I'd have a guess and say
that it won't, though, as I have already determined that it is pretty
poor at detecting polymorphic viruses.

Instead of damning ClamAV out of hand, though, I prefer to take the
optimistic view that this is something that will be addressed in the
near future. There are apparently many organizations using ClamAV to
scan mail on their mail servers. I see no reason why the kind of factors
that drove the development of the Apache web server or the Linux OS will
not eventually lead to improvements in ClamAV's current weak areas.

AV bloat is primarily due to HTML interfaces and other similarly idiotic
"innovations".

Those things are certainly a major cause of bloat, but I recall that the
growth in the number of viruses caused the developers of DOS scanners to
have to use DOS extenders to overcome the 640K limit, so I guess the
number of viruses must also have an impact on memory usage.

 

[Frederic Bonroy]

Julian a écrit :

I don't have a sample, so I can't tell you. I'd have a guess and say
that it won't, though, as I have already determined that it is pretty
poor at detecting polymorphic viruses.

Probably, yes.

Instead of damning ClamAV out of hand, though, I prefer to take the
optimistic view that this is something that will be addressed in the
near future. There are apparently many organizations using ClamAV to
scan mail on their mail servers.

Shouldn't they wait until Clamav becomes a real alternative to
professional scanners, if it happens someday? Don't you think it's
dangerous to rely on Clamav in its current state?