MICHAEL said:
It's about more than one or two tests. An AV company can
achieve the 100% award once and display that for many months
even though they may not put themselves up to be tested
again for a year.
http://www.virusbtn.com/vb100/archive/results?display=vendors
VB100% Results Overview: Grisoft (AVG)
13 Success / 21 Failure / 13 No Entry
VB100% Results Overview: Trend Micro
14 Success / 7 Failure / 26 No Entry
VB100% Results Overview: Alwil (Avast)
18 Success / 19 Failure / 10 No Entry
VB100% Results Overview: F-Secure
23 Success / 12 Failure / 12 No Entry
VB100% Results Overview: Computer Associates (eTrust/InoculateIT)
24 Success / 11 Failure / 12 No Entry
VB100% Results Overview: McAfee
27 Success / 18 Failure / 2 No Entry
VB100% Results Overview: Symantec
33 Success / 6 Failure / 8 No Entry
VB100% Results Overview: Sophos
33 Success / 12 Failure / 2 No Entry
VB100% Results Overview: Kaspersky
34 Success / 13 Failure / 0 No Entry
VB100% Results Overview: Eset (NOD32)
39 Success / 3 Failure / 5 No Entry *23 Success in a row- the longest streak of any AV.*
I will say, AVG has gotten 100% 4 times in row,
and 10 out of the last 11 that they have had their
AV tested. They had a horrendous product from
1998 until about 2004. Avast has similar results.
They did terrible from about 1998 until 2003. They
have now achieved 100% 9 times in a row. Remember,
some AV vendors participate more often than others.
They only have to achieve 100% once every 12 months
in order to use the VB 100% award logo. The tests are
actually rather strict- no "in the wild" misses and no false
positives.
One last important thing to remember- *none* of these
AV vendors have been tested on Vista, yet. The testing
schedule indicates AVs will be tested on Vista Feb. 2007.
More on the testing procedures:
http://www.virusbtn.com/vb100/about/100procedure.xml
VB 100% test procedures
VB 100% award denotes that the product in question showed, in its default mode, 100 per cent
detection of In the Wild test samples and no false positives in a selection of clean files.
For on-demand scanning of files, detection is considered to be a note in the product log file
that the file is infected or very likely so. For on-demand scanning of boot sector viruses, a
notification or log file entry is required.
For on-access scanning the matter is a little more confusing, since the best method of
testing - executing all files and using the results from this activity - is clearly
impractical. Detection is thus judged by a product denying access to an infected file when the
file is opened for writing.
For boot sector on-access scanning a visible notification or log file entry is required. In
this case denial of access is not a useful guide to detection since the VB boot sector test
floppies are all blank as far as file contents are concerned. Since denial of access is likely
to show a blank disk as the only detectable effect, this is not particularly useful. The
addition of extra files to the disk for use in deciding whether access has been denied was
decided against, for in past testing some products were only able to detect a boot sector virus
on a floppy containing other files - a situation which would be apparent only with the use of
disks in their current state.
Products which cannot be cajoled into producing reasonable logs on demand are checked by
setting the product to delete and/or disinfect. The files are then scanned until no more
detections are present, if necessary manually noting those files which are detected as infected
but are not deleted or disinfected. Disinfected files are removed from the test set by use of
CRC checking, and those files left in the test set are considered to be misses.
Near misses
There remains ample opportunity for products to miss detection, in our tests, of files which
they are perfectly able to detect - why? Of the many potential answers, two are most likely.
First, there are the matters of default extension lists, a common area for failure over the
years, in which products have failed to gain VB 100% awards because the default extension lists
did not include possible extensions for In the Wild viruses. In most cases these
extension-based problems are easily solved by an administrator adding extensions to the default
list. We could perform these changes prior to testing. We feel, however, that our readers are
better served if they know that they have to do this, than if we scan all files regardless of
extension.
Another example of why some products miss out on VB 100% awards, is where certain files are not
scanned directly on-access. The usual assumption by the product developers is that the files
will be scanned when passed on to an application which makes use of them. At the most common
level this covers such objects as ZIP files, which are often not scanned until unzipped and EML
files, which are not scanned until individual mails are pulled from within. From a developer's
point of view these choices make sense in that leaving objects unscanned until use creates
fewer overheads. The chance of infection on a protected machine is not increased, since
scanning will occur before code execution. Such treatment of objects does, however lead to
misses under the VB 100% testing methodology.
Three chances
Each product may be tested up to three times on two different test machines. Should any product
fail to work after three attempts the testing process will be aborted for that product.
VB 100% award
A VB 100% award means that a product has passed our tests, no more and no less. The failure to
attain a VB 100% award is not a declaration that a product cannot provide adequate protection
in the real world if administered by a professional. We would urge any potential customer, when
looking at the VB 100% record of any software, not simply to consider passes and fails, but to
read the small print in the reviews.
-Michael
