AV products tested vs 50K virii

[kurt wismer]

Nick has sort of defined his infamous term "crud" through the years. I
know he includes the remnants of botched disinfections, for just one
example. You and I have discussed this before. Remember when I pointed
out how F-Prot with the /collect (for virus collection tests) switch
on will alert on some "crud" files I sent to you? Some other crud in
typical vxer collections include infected boot sector images, object
files, and even .ASM text files. Of course I've never seen an av alert
on a text file regardless of the file extension. I doubt if any av
product is that stupid.

But I view all malware as crud whether the crap is viable or not. I'm
not discriminating

Insofar as av vendors keeping up with crud as well as viable samples,
it seems that KAV and McAfee do a pretty good job of it. Others like
F-Prot also "know about" many vxer crud files but won't alert unless
asked to. I do prefer Frisk's approach.

i've seen f-prot false alarm on a text file... in fact the current
version false alarms on a text file on my drive... the filename is
taglines.bak and it is, as you might guess, a backup copy of my
taglines file from back when i used taglines in fidonet...

f-prot reports it's infected with a new or modified variant of
trivial... i suspect because somewhere in the middle of the file is the
eicar string (i was mostly only interested in the virus related forums
in fidonet)... thankfully f-prot and other scanners no longer alert on
"don't scan this tagline, it has the 141$FLu..."...

 

[kurt wismer]

.vbs?

or .html for that matter...

hey, i've got one...

..bat ....

 

[kurt wismer]

what a wonderful argument in favour false positives...

False positives are quite a different matter. I have tried to raise
the issue off and on through the years here concerning the question
"Does alerting on crud constitute a false positive?". Nobody ever
responded.[/QUOTE]

that seems unlikely... where was i? i would have most certainly
answered "yes it's a false positive"...

So far as I can tell, the av professionals seem to take the
tac that alerting on crud does _not_ constitute a false postive. So
deifne your terms.

the term is quite well defined... a false positive is an alert that
says "X is here" when it is not...

the 'crud' used in the test can also be called 'scanner fodder' -
perhaps that's the term you wanted defined...

 

[kurt wismer]

Yes, my tests are not 100% flawless. But i also believe that most of the files i have in my collection
are not garbage, as some mention. If they were garbage indeed,
why do most respectable av software detect them as malware?

read the rest of the thread... that is explained...

Is this, too, my fault?

no, bad experimental design is your fault...

Or is it that, certain av companies
tend to add many more stuff that before, just to have a huge number of detectable viruses?

no, that crud doesn't count towards the number of viruses they
detect... that crud is there so that they don't look bad when people
like you do tests...

And, isn't
it a good way to increase sales?Or is that also my fault? And, when it comes to the bottom of this, how can
i trust ANY av software, if i really cannot tell how many REAL viruses it detects..??

it would seem you cannot tell how many REAL viruses you have, so how
are you to determine how many REAL viruses a product detects?

Anyway, as some of you know, i have had quite bitter comments about this last test. So, i decided to start
learning more things about viruses, hoping that sometime soon i will
be able to maintain a really good collection,

read "Analysis and Maintenance of a Clean Virus Library"
(http://www.virusbtn.com/old/OtherPapers/VirLib/)

as for "soon", i wouldn't hold my breath (because i've already read the
paper)...

without any garbage files in it. What i have seen lately,
is that most av'ers do not want to hear about us, "vxers", and generally do
not want anything to do with us. I ask you, now, if a guy like me wants and tries to learn all the stuff that
professional avers know, how can he accomplish it?

shameless plug - http://k_wismer.tripod.com/papers.htm ...

 

[null]

that seems unlikely... where was i?

So now I'm supposed to try to find all those ancient history posts
just to prove I'm not dreaming this up? Forget it Kurt. I'm neither a
dreamer nor a lier as you imply.

i would have most certainly
answered "yes it's a false positive"...

Which tests use them for false positive testing?

the term is quite well defined... a false positive is an alert that
says "X is here" when it is not...

That's a worthless definition without defining X.

the 'crud' used in the test can also be called 'scanner fodder' -
perhaps that's the term you wanted defined...

The crud used in which test? What's scanner fodder? I'm guessing that
you might mean botched disinfections. But "crud" as Nick uses the term
is far more inclusive.


Art
http://www.epix.net/~artnpeg

 

[Nick FitzGerald]

I guess i should explain some things over here as well, regarding the
test i performed. ...

Why?

Are you feeling guilty and trying to assuage that feeling?

You _need_ not explain anything. In fact, I doubt you can explain very
much at all as we have seen many useless, ill-informed, mis-directed
"tests" just like this over and over for most of the last 15 years.

The fact is, we know more about your tests than you do, which is the
worst indictment that can be made about of them...

... I once again admit that i still miss alot of knowledge about
viruses, compared to professional av'ers.

Good.

Then here's a deal for you. Before even contemplating doing another of
your shitty, worthless tests spend some time searching out some of the
very good articles out there about how to do virus detection tests.
You will recognize them because they are not written by pimply-faced,
ansgt-ridden moroniuc teenagers or other Vxers and they will be replete
with examples of bad testing processes that are found throughout your
testing methodology.

You will also want to look for the good articles on developing a top-
flight virus (or, these days, "malware") collection. Again, you will
find that the articles worth reading are all _NOT_ written by VXers...

Still, i will not accept people calling me stupid or moron or other
"specially chosen" nicknames, just because i do not know as much as they do.

So, lump it.

You are a moron because despite the voluminous, accurate and deserved
discrediting of your tests you still try to defend them.

Yes, my tests are not 100% flawless. ...

Would you like to try that again, without the marketing spin?

A statement something like "Yes, my tests are 100% flawed" would be much
more accurate and indicative that you have started to understand something
ratehr important...

... But i also believe that most of the files i have in my collection
are not garbage, as some mention. If they were garbage indeed, why do
most respectable av software detect them as malware? ...

What proportion is this "most" you are talking about?

80%

90%

92%

95%

From a very quick look at your results, I'd be surprised if much more
than 90% are "valid" malware samples (and it may be somewhat lower if
we take out boot infector sector dumps and so on...). That may seem
like "most" to you, but that leaves a pretty wide error margin in
your results...

... Is this, too, my fault? Or is it that, certain av companies
tend to add many more stuff that before, just to have a huge number
of detectable viruses? ...

They don't add them to boost the "X" in "we detect X viruses" claims.

They detect them for one of two or three reasons. Either their detection
methods for that type of file/malware/etc are rather poor or their lab
staff did a poor job analysing that malware or they specifically added
detection of the crud because such files (and I mean, precisely the same
files) have often been used in shite tests such as yours (in the latter
case, whether such detections are added to "protect themselves" against
incompetent testers or to cynically boost their product against more
honest competitors when compared in incompetent tests is something you'll
have to decide on a case by case basis).

Also, you say that most products detect most of these files. Do you
actually understand what it means when F-PROT detects a PE as "corrupted
executable" an old, would-be DOS virus as "intended", a macro in a Word
document as "inactive" and so on? It means that there's nothing of
actual concern present -- sure it is a partial attempt to protect the
product against idiots like you, but note that it these are really
"anti-detections" rather than "real" detections. Similar comments apply
to many other "protective anti-detections" used in other products too,
yet you continue to see these as evidence of the quality of your tests.
If you still do not see the problem, then "moron" is clearly not strong
enough a term...

... And, isn't it a good way to increase sales?Or is that also my fault?

Huh?? What has that to do with the price of fish?

... And, when it comes to the bottom of this, how can
i trust ANY av software, if i really cannot tell how many REAL viruses
it detects..??

There you hit the nail right on the head, though not for the reason I
suspect you think.

You are dead right -- _YOU_ are incapable of telling how many real viruses
any AV product detects. However, that is not because of a flaw in the AV
software but because of several obvious flaws in your wetware; flaws you
seem rather unwilling to address and correct.

Anyway, as some of you know, i have had quite bitter comments about this
last test. ...

Really???

Well that's a surprise -- NOT!

Could it be because it is, without a doubt, the biggest load of shite that
has tried to pass itself of as an AV test for several years?

... So, i decided to start
learning more things about viruses, hoping that sometime soon i will
be able to maintain a really good collection, without any garbage files
in it. ...

Well, that would certainly be a good start. However, I doubt you will be
able to go on to your next test "sometime soon" -- starting this late it
will take you more than a few weeks or months to build such a collection.

... What i have seen lately, is that most av'ers do not want to hear
about us, "vxers", and generally do not want anything to do with us. ...

Have you considered that this may be because most VXers are almost totally
lame and despite their own best efforts seldom are doing much more than
mocking the processes and procedures of real AV research?

Also, the really, really bad tests -- such as yours -- are either done by
VXers or enabled by the public VX sites.

... I
ask you, now, if a guy like me wants and tries to learn all the stuff that
professional avers know, how can he accomplish it?

By using his intellect.

You do have a brain, yeah??

Try using it. Despite running a fairly closed shop, there is a fairly
extensive literature out there describing everything you need to know
about designing and running a comprehensive AV tests.

To sum up, i 'd like to say that i performed the test using the best
methodology i could get, based on my current knowledge. Yet, that was not
enough, i know it and i do not deny it.

Odd that you tried to do something that was clearly technical without even
finding the most basic of information about it available through some
simple Googling...

I really try to catch up on the info i miss and i hope that in future
tests my methodology will be better. That is what i aim at anyway.
Thanks for your support and comments, good and bad

You're welcome and good luck...

 

[kurt wismer]

On Thu, 08 Jan 2004 20:22:51 +0100, Frederic Bonroy
[snip]

That depends on the kind of crud we are talking about, no? Crud can
be "working crud" (in that it's able to cause damage of some sort) or it
can be "crud crud" that doesn't work at all. Why make any efforts to
detect the latter?


Why not? I don't care to have that crud on my hard drive. Do you? So
why not alert on it?

what a wonderful argument in favour false positives...


False positives are quite a different matter. I have tried to raise
the issue off and on through the years here concerning the question
"Does alerting on crud constitute a false positive?". Nobody ever
responded.

that seems unlikely... where was i?

So now I'm supposed to try to find all those ancient history posts
just to prove I'm not dreaming this up? Forget it Kurt. I'm neither a
dreamer nor a lier as you imply.[/QUOTE]

i implied neither... i suggested that you were simply wrong...

Which tests use them for false positive testing?

something doesn't have to be used in a false positive test to qualify
as a false positive...

that said, crud is not malware, crud is not viral, and crud is not
wormlike - if it were any of these it should be referred to as one of
these instead... crud is garbage, crud is that stuff that scanners
alert on but shouldn't - all false positive tests have used 'crud'....

That's a worthless definition without defining X.

?? what are you talking about?

if something says "NYB is here" when it is not then it is a false
positive... likewise if something says "Osama bin Ladin is here" when
he is not it is also a false positive...

X is perfectly fine as a variable noun...

The crud used in which test?

the test in question, the test this thread is about...

What's scanner fodder?

random stuff thrown at a scanner for no other reason than the fact that
the scanner will alert on it (erroneously)...

I'm guessing that
you might mean botched disinfections.

no, scanner fodder is not "botched disinfections"... it may include
botched disinfections but is not limited to that set...

But "crud" as Nick uses the term
is far more inclusive.

same here...

 

[Frederic Bonroy]

that said, crud is not malware, crud is not viral, and crud is not
wormlike - if it were any of these it should be referred to as one of
these instead... crud is garbage, crud is that stuff that scanners
alert on but shouldn't - all false positive tests have used 'crud'....

What about a damaged Trojan horse that doesn't behave exactly like its
undamaged version but still manages to do something unwanted. Would you
consider that to be crud or not?

Or a damaged virus that doesn't replicate anymore but still manages to
do something unwanted...

 

[kurt wismer]

What about a damaged Trojan horse that doesn't behave exactly like its
undamaged version but still manages to do something unwanted. Would you
consider that to be crud or not?

Or a damaged virus that doesn't replicate anymore but still manages to
do something unwanted...

most programs, regardless of type, manage to do something unwanted when
they are damaged... it is the nature of damage to not generally produce
constructive results...

if, however, it is being reported as damaged and still works as
intended (or perhaps more accurately, works the same or almost the same
as the undamaged version) then it is not really damaged but rather a
new or modified variant of . . . that the vendor hasn't had a chance to
analyze yet...

 

[null]

(e-mail address removed) wrote:
On Thu, 08 Jan 2004 20:22:51 +0100, Frederic Bonroy

[snip]

That depends on the kind of crud we are talking about, no? Crud can
be "working crud" (in that it's able to cause damage of some sort) or it
can be "crud crud" that doesn't work at all. Why make any efforts to
detect the latter?


Why not? I don't care to have that crud on my hard drive. Do you? So
why not alert on it?

what a wonderful argument in favour false positives...


False positives are quite a different matter. I have tried to raise
the issue off and on through the years here concerning the question
"Does alerting on crud constitute a false positive?". Nobody ever
responded.

that seems unlikely... where was i?

So now I'm supposed to try to find all those ancient history posts
just to prove I'm not dreaming this up? Forget it Kurt. I'm neither a
dreamer nor a lier as you imply.

i implied neither... i suggested that you were simply wrong...

You certainly did imply that I'm lying or dreaming. Read what you
wrote. You claimed that my asking the questions here in the past is
"unlikely".

And if you claim I'm wrong about something, please be specific. I have
no idea what you believe I'm wrong about.

something doesn't have to be used in a false positive test to qualify
as a false positive...

What kind of gibberish is that? Scanner X is accused of being a super
crud detector. Yet published "quality" tests all show a extemely low
false positive rate. Huh? What? Hey, there is no meaning then to the
"quality" tests is there? Seems obvious to me that "quality" tests
don't include "crud" in their false positive testing. See my point?
See the thrust of my questioning?

that said, crud is not malware, crud is not viral, and crud is not
wormlike - if it were any of these it should be referred to as one of
these instead... crud is garbage, crud is that stuff that scanners
alert on but shouldn't - all false positive tests have used 'crud'....

Oh? Then why do the "super crud detectors" always show quite low and
competitive false alarms rates in the so called "quality independent
tests"? The fact is that the super crud detectors manage to avoid the
kind of false positives that really count. I can run them year after
year on my PCs and they never produce a "real" false alert (on
perfectly good files). So what's the big deal about crud detectors?
What's so damn bad about them? See my point and my issues here?

?? what are you talking about?

if something says "NYB is here" when it is not then it is a false
positive...

Not really. It may be a misidentification.

likewise if something says "Osama bin Ladin is here" when
he is not it is also a false positive...

Not if it's a viable NYB instead.

X is perfectly fine as a variable noun...

LOL! Maybe a bit "too variable"

the test in question, the test this thread is about...

I doubt if all his 58,000 samples are crud.

random stuff thrown at a scanner for no other reason than the fact that
the scanner will alert on it (erroneously)...

That's just one kind of "crud". There are many others, as have been
discussed here in this thread and in the past.


Art
http://www.epix.net/~artnpeg

 

[kurt wismer]

(e-mail address removed) wrote:

On Thu, 08 Jan 2004 22:35:44 -0500, kurt wismer <[email protected]>

(e-mail address removed) wrote: [snip]
False positives are quite a different matter. I have tried to raise
the issue off and on through the years here concerning the question
"Does alerting on crud constitute a false positive?". Nobody ever
responded.

that seems unlikely... where was i?


So now I'm supposed to try to find all those ancient history posts
just to prove I'm not dreaming this up? Forget it Kurt. I'm neither a
dreamer nor a lier as you imply.

i implied neither... i suggested that you were simply wrong...

You certainly did imply that I'm lying or dreaming. Read what you
wrote. You claimed that my asking the questions here in the past is
"unlikely".

no, i claimed that you asking the questions as posed above and never
getting a response was 'unlikely'...

And if you claim I'm wrong about something, please be specific. I have
no idea what you believe I'm wrong about.

either you didn't pose the questions exactly the same way in the past
or there were in fact responses or both... i suspect the former...

then there's the issue of you saying you did it here - here is
alt.comp.anti-virus which in the grand scheme of things is a relatively
new group... maybe you did it elsewhere...

What kind of gibberish is that?

just as i don't have to be included in the human genome project to
qualify as a human, a particular false positive or even a particular
type of false positive does not have to be included in a false positive
test to qualify as a false positive...

Scanner X is accused of being a super
crud detector. Yet published "quality" tests all show a extemely low
false positive rate. Huh? What? Hey, there is no meaning then to the
"quality" tests is there? Seems obvious to me that "quality" tests
don't include "crud" in their false positive testing. See my point?
See the thrust of my questioning?

i suspect you misunderstand the nature of false positive tests... but
more than that, i think this has nothing to do with whether or not
something is a false positive... being used in a false positive test is
not the defining quality of a false positive... ergo, by extension the
results of published false positive tests have nothing to do with
whether or not something is a false positive...

Oh? Then why do the "super crud detectors" always show quite low and
competitive false alarms rates in the so called "quality independent
tests"?

because false positive tests don't use 'crud' exclusively...

The fact is that the super crud detectors manage to avoid the
kind of false positives that really count. I can run them year after
year on my PCs and they never produce a "real" false alert (on
perfectly good files). So what's the big deal about crud detectors?
What's so damn bad about them? See my point and my issues here?

you seem to want to separate crud from so-called 'real' false positives...

the point you overlook is that crud *is* a real false positive...

Not really. It may be a misidentification.

sophistry... a false positive and a false negative do not cancel each
other out... if there were some other virus there instead of NYB then
the detector compounded it's error...

Not if it's a viable NYB instead.

no, no, and again i say no...

the statement "Osama bin Ladin is here" is a positive statement (had i
used "isn't" instead of "is" it would be a negative statement)... if
Osama bin Ladin isn't actually here then the statement "Osama bin Ladin
is here" is false, by definition...

LOL! Maybe a bit "too variable"

no, not at all... the concept of a false positive very general and
widely used... it boggles the mind that you would need it defined...

I doubt if all his 58,000 samples are crud.

indeed... statistically, with 58,000 samples, there's a high
probability that at least one of them was a real virus...

That's just one kind of "crud". There are many others, as have been
discussed here in this thread and in the past.

to my mind, 'random stuff' is fairly all inclusive in it's generality...

 

[virusp__at_]

Why?

Are you feeling guilty and trying to assuage that feeling?

You _need_ not explain anything. In fact, I doubt you can explain very
much at all as we have seen many useless, ill-informed, mis-directed
"tests" just like this over and over for most of the last 15 years.

The fact is, we know more about your tests than you do, which is the
worst indictment that can be made about of them...


Good.

Then here's a deal for you. Before even contemplating doing another of
your shitty, worthless tests spend some time searching out some of the
very good articles out there about how to do virus detection tests.
You will recognize them because they are not written by pimply-faced,
ansgt-ridden moroniuc teenagers or other Vxers and they will be replete
with examples of bad testing processes that are found throughout your
testing methodology.

You will also want to look for the good articles on developing a top-
flight virus (or, these days, "malware") collection. Again, you will
find that the articles worth reading are all _NOT_ written by VXers...


So, lump it.

You are a moron because despite the voluminous, accurate and deserved
discrediting of your tests you still try to defend them.

You are dead right -- _YOU_ are incapable of telling how many real viruses
any AV product detects. However, that is not because of a flaw in the AV
software but because of several obvious flaws in your wetware; flaws you
seem rather unwilling to address and correct.


Really???

Well that's a surprise -- NOT!

Could it be because it is, without a doubt, the biggest load of shite that
has tried to pass itself of as an AV test for several years?

Nick FitzGerald

I see you continue to swear .. sorry pal, although you seem to know many things about viruses,
i realize my manners are far better than yours.

There are some things respected in REAL world far more than vx, like a person's manners,
but i see you don't give a dime about that.

Good luck to you too, trying to become a polite and civilized person.

 

[null]

(e-mail address removed) wrote:


On Thu, 08 Jan 2004 22:35:44 -0500, kurt wismer <[email protected]>

(e-mail address removed) wrote: [snip]
False positives are quite a different matter. I have tried to raise
the issue off and on through the years here concerning the question
"Does alerting on crud constitute a false positive?". Nobody ever
responded.

that seems unlikely... where was i?


So now I'm supposed to try to find all those ancient history posts
just to prove I'm not dreaming this up? Forget it Kurt. I'm neither a
dreamer nor a lier as you imply.

i implied neither... i suggested that you were simply wrong...

You certainly did imply that I'm lying or dreaming. Read what you
wrote. You claimed that my asking the questions here in the past is
"unlikely".

no, i claimed that you asking the questions as posed above and never
getting a response was 'unlikely'...

I see. Sorry for the misunderstanding.

either you didn't pose the questions exactly the same way in the past
or there were in fact responses or both... i suspect the former...

then there's the issue of you saying you did it here - here is
alt.comp.anti-virus which in the grand scheme of things is a relatively
new group... maybe you did it elsewhere...

It was certainly on alt.comp.virus several years ago. I remember Axel
Pettinger suggested that I push for or do myself some sort of "crud"
evaluation tests of scanners

just as i don't have to be included in the human genome project to
qualify as a human, a particular false positive or even a particular
type of false positive does not have to be included in a false positive
test to qualify as a false positive...

Indeed but I still believe that crud detection should be tested for
separately.

i suspect you misunderstand the nature of false positive tests...

Perhaps so. I've never tried to delve into the nature of samples that
are used ... on what basis they are selected, etc. I've sort of
assumed that clean legit files that are found on typical PCs are used.
No?

But
more than that, i think this has nothing to do with whether or not
something is a false positive... being used in a false positive test is
not the defining quality of a false positive... ergo, by extension the
results of published false positive tests have nothing to do with
whether or not something is a false positive...

because false positive tests don't use 'crud' exclusively...

Oh? By "exclusively" you imply that some crud is purposely included??
If so then why do alleged super crud detectors also manage to score
the lowest false positive rates (speaking generally)? Now, I'm too
lazy to go back and dig up all the independent test results I've
looked at over the years which have led me to this conclusion.
Probably it has been mainly the VTC tests over the years that have
shown this ... but I think av-test.org tests show the same sort of
thing. Maybe VB uses some crud in their false positive testing which
indicates differently? I dunno.

you seem to want to separate crud from so-called 'real' false positives...

I most certainly do!!!

the point you overlook is that crud *is* a real false positive...

I'm not overlooking it I'm questioning it, debating it, and arguing
against that definition. I thought that was clear enough.

Note that I'm _not_ suggesting that crud detection should be ignored.
I just think it's in a special category. And that category can be
tested independently of "real" false positive tests. In my mind,
"real" false positive tests should be geared to what users (and
vendors) are really concerned about ...annoyances, lost time and money
due to scanners false alarming on perfectly legit files.

I'd like to see crud detection tests conducted, and pressures exerted
on vendors to report clearly and distinguish between crud and actual
infections.


Art
http://www.epix.net/~artnpeg

 

[Nick FitzGerald]

<virusp__at_@_virus.gr> replied to me:

I see you continue to swear .. sorry pal, although you seem to know many
things about viruses, ...

ROTFLMAO... "seem to"...

Kid -- I've forgotten more useful information about viruses than it appears
you are capable of knowing.

Oh, and I do "condescending" much better than you too...

... i realize my manners are far better than yours.

Excuse me???

You walk into a complex, technical industry you know nothing about, decide
that not knowing anything about it cannot be any kind of impediment to
testing the quality of the products of that industry, set out to run your
tests without even trying to learn if there are "good", "poor" and
"outright bad" tests or how other tests rated at different poits on that
scale differ in the way they are done or why the good ones do the "strange"
(to your eye, after the fact) things thay do. Therefore, you inevitably
make every dipshit, dimbulb moron error in the book _just like all the
dozens of equivalently clueless and grossly unfit to the task lusers before
you who have done the same_, who you would know about and (some of) whose
errors you would realize you were about to repeat had you spent five
minutes searching Google _before you started_.

In what way is that not entirely inexcusably rude?

Huh???

However, not content with that level of rudeness, you then find some moron
journalists who clearly understand even less than you about virus detection
testing (it would be a challenge, but you managed to succeed there) and
convince them to publish the results of what you laughingly call a "test".

In what way is that not immensely ruder still?

And now, when some of us brusquely and matter-of-factly point out your
voluminous errors, all you can do is whinge is that we are being rude to
you??

Sonny, you need to grow up. For one thing I was not rude in my previous
post -- I certainly was not obsequiously polite, but I wasn't rude. I
bluntly pointed out some of the areas of gravest error in your approach
and interprpetation of the results. Believe me, I have not even started
being rude yet to you yet bucko, but when I do, you'll know it...

There are some things respected in REAL world far more than vx, like a
person's manners, but i see you don't give a dime about that.

Snot-rags like you come and go every year or so. As the purveyors of such
"tests" -- of which we have seen many through the years -- have proven to
be reprehensible shitbags, grossly ignorant beyond redemption, completely
ineducable about "good" testing processes and generally utterly rude in
their approach and lack of concern for "fixing" the damage they do, some
of us don't bother trying to be polite to the new examples of such slime
as they come to our attention through their unbridled rudeness. They
always skulk away without so much as an "I'm sorry" too, so why should we
be especially polite to them?

You have shown yourself to be a typical sample, so are being treated
somewhat better than you treat the industry though not especially politely
-- I save that for people who are polite to me -- and as history strongly
suggests you deserve.

Good luck to you too, trying to become a polite and civilized person.

I am a polite and civilized person, not that I care for the opinion in
such matters of a jumped-up Cretan sheep-shagger.

Your worst error is not in judging me by my actions over this issue but in
failing to note that you are being treated as your actions (and possibly
motivations) deserve. These, of course, are things you could have "fixed"
from the outset of your escapade by spending a few minutes researching
whatever the **** it was you thought you were doing. That, however, would
have required some minscule intellect on your part, so your failure to do
it shows you are an abject moron. You are also clearly far to stupid to
understand the logic of this and may well try to argue the toss with me
(though I detect a hint of huffiness about your response and you may just
retire, incorrectly feeling you have withdrawn after asserting what you
strangely see as some moral highground...).

And, in case you truly are the unduly sensitive type (in which case nothing
about the AV business will suit you) and are not sure whether I've started
being rude yet, I haven't, but I am warming to the task should you wish to
continue addressing me rather than the errors I've started pointing out in
your tests...

 

[kurt wismer]

(e-mail address removed) wrote: [snip]

And if you claim I'm wrong about something, please be specific. I have
no idea what you believe I'm wrong about.

either you didn't pose the questions exactly the same way in the past
or there were in fact responses or both... i suspect the former...

then there's the issue of you saying you did it here - here is
alt.comp.anti-virus which in the grand scheme of things is a relatively
new group... maybe you did it elsewhere...

It was certainly on alt.comp.virus several years ago. I remember Axel
Pettinger suggested that I push for or do myself some sort of "crud"
evaluation tests of scanners

i googled... i found the thread in question... from my reading of it,
axel considers such testing to be a special kind of false positive
testing - ergo crud constitutes a false positive to him, ergo you got
an answer to the question you said never got answered...

msgid <[email protected]> if you care to
check...

it also seems from the discussion that you also considered them false
positives at that time...

[snip]

Indeed but I still believe that crud detection should be tested for
separately.

nonsense... a false positive test (a good one) should use a large and
statistically representative sample of files that contain neither
viruses, nor worms, nor trojans, nor any of the other types of malware
that anti-virus products are now trying to detect...

since crud is not viral, not worm like, not a trojan, etc ... it should
be included in such a test...

Perhaps so. I've never tried to delve into the nature of samples that
are used ... on what basis they are selected, etc. I've sort of
assumed that clean legit files that are found on typical PCs are used.
No?

realistically, it's impossible to define what is on a typcial pc...
it's more practical to deal with the complement of the set of files
that would constitute a true positive... anything and everything that
isn't viral/worm/trojan/etc goes into the pool and samples selected at
random...

[snip]

Oh? By "exclusively" you imply that some crud is purposely included??
If so then why do alleged super crud detectors also manage to score
the lowest false positive rates (speaking generally)?

because crud represents a small fraction of the total population of
non-viral/worm/trojan/etc files... by extension, there should only be a
relatively small number of crud files in a false positive test...

[snip]

I most certainly do!!!


I'm not overlooking it I'm questioning it, debating it, and arguing
against that definition. I thought that was clear enough.

then what you're really debating is the meaning of 'crud'...

i think it's clear from the derrogatory term that 'crud' constitutes a
class of files that get detected for no legitimate reason... they don't
represent any kind of real threat... they aren't viruses, they aren't
worms, they aren't trojans, etc...

why then, should we consider their detection something positive,
beneficial, or true...

Note that I'm _not_ suggesting that crud detection should be ignored.
I just think it's in a special category. And that category can be
tested independently of "real" false positive tests. In my mind,
"real" false positive tests should be geared to what users (and
vendors) are really concerned about ...annoyances, lost time and money
due to scanners false alarming on perfectly legit files.

some of the things that fall into the crud category are perfectly legit
files... i don't just mean that they're non-threats, i mean they're
legitimate applications or parts thereof...

if you think about where the large collections of crud are and how they
come to be it seems obvious that real false positives are going to be
thrown into a virus collectors collection because quite often they
haven't the skill, time, patience to weed out the real threats from the
false alarms - besides, a bigger collection is better...

I'd like to see crud detection tests conducted, and pressures exerted
on vendors to report clearly and distinguish between crud and actual
infections.

then you misunderstand what crud is... this is a chicken and egg
scenario but consider this - if you think av companies went through
collectors collections and said "this isn't a virus, lets add it to our
crud detection" then ask yourself how it got into the collector's
collection in the first place... somebody, somewhere, thought it was a
virus, which suggests some program called it a virus, ergo it was
already getting detected falsely (assuming it's crud and not really a
virus)...

so - for an anti-virus to differentiate between real viruses and crud,
the anti-virus would have to know when it's alerting falsely... and if
it could do that, why would it bother issueing false alerts at all?

 

[null]

(e-mail address removed) wrote: [snip]
And if you claim I'm wrong about something, please be specific. I have
no idea what you believe I'm wrong about.

either you didn't pose the questions exactly the same way in the past
or there were in fact responses or both... i suspect the former...

then there's the issue of you saying you did it here - here is
alt.comp.anti-virus which in the grand scheme of things is a relatively
new group... maybe you did it elsewhere...

It was certainly on alt.comp.virus several years ago. I remember Axel
Pettinger suggested that I push for or do myself some sort of "crud"
evaluation tests of scanners

i googled... i found the thread in question... from my reading of it,
axel considers such testing to be a special kind of false positive
testing - ergo crud constitutes a false positive to him, ergo you got
an answer to the question you said never got answered...

Not really. I was never satisified with the answers/non-answers I was
getting then. I have no idea why you seem to think that a response or
opinion from someone ... even as astute as Axel, would put the
question to rest in my mind. I've never seen any kind of thorouh-going
discussion on this issue.

msgid <[email protected]> if you care to
check...

it also seems from the discussion that you also considered them false
positives at that time...

But then as now not a "real" false positive

[snip]

Indeed but I still believe that crud detection should be tested for
separately.

nonsense... a false positive test (a good one) should use a large and
statistically representative sample of files that contain neither
viruses, nor worms, nor trojans, nor any of the other types of malware
that anti-virus products are now trying to detect...

since crud is not viral, not worm like, not a trojan, etc ... it should
be included in such a test...

But what kind of crud? For example, I recall at the time, some of the
crud in vxer collections I downloaded were object files and BSIs. How
many typical users have such files on their PCs? Now, some users may
have the results of botched disinfections. Maybe a very small
percentage of false postive test samples could include these. But it's
nonsense to include crud that most users will never realistically
experience.

realistically, it's impossible to define what is on a typcial pc...
it's more practical to deal with the complement of the set of files
that would constitute a true positive... anything and everything that
isn't viral/worm/trojan/etc goes into the pool and samples selected at
random...

I strongly disagree. You're not being practical or realistic at all.

[snip]

Oh? By "exclusively" you imply that some crud is purposely included??
If so then why do alleged super crud detectors also manage to score
the lowest false positive rates (speaking generally)?

because crud represents a small fraction of the total population of
non-viral/worm/trojan/etc files... by extension, there should only be a
relatively small number of crud files in a false positive test...

You speak of "should" and it appears that you're expressing your
opinion. I'm most interested in what is actually done, and for that
I've never gotten a definitive answer.

[snip]

I most certainly do!!!


I'm not overlooking it I'm questioning it, debating it, and arguing
against that definition. I thought that was clear enough.

then what you're really debating is the meaning of 'crud'...

i think it's clear from the derrogatory term that 'crud' constitutes a
class of files that get detected for no legitimate reason... they don't
represent any kind of real threat... they aren't viruses, they aren't
worms, they aren't trojans, etc...

why then, should we consider their detection something positive,
beneficial, or true...

Don't ask me. Ask someone who thinks alerting on them as if they are
actually "for real" is something positive, beneficial or true. As I've
said, the problem with many scanners is that they don't identify the
crud for what it is. It's a reporting problem to some extent.

some of the things that fall into the crud category are perfectly legit
files... i don't just mean that they're non-threats, i mean they're
legitimate applications or parts thereof...
Example?

if you think about where the large collections of crud are and how they
come to be it seems obvious that real false positives are going to be
thrown into a virus collectors collection because quite often they
haven't the skill, time, patience to weed out the real threats from the
false alarms - besides, a bigger collection is better...

Real false postives in a virus collection? I have no idea what you
have in mind there.

then you misunderstand what crud is...

Oh? I've certainly seen enough of it. Why do you say I don't
understand it?

this is a chicken and egg
scenario but consider this - if you think av companies went through
collectors collections and said "this isn't a virus, lets add it to our
crud detection" then ask yourself how it got into the collector's
collection in the first place... somebody, somewhere, thought it was a
virus, which suggests some program called it a virus, ergo it was
already getting detected falsely (assuming it's crud and not really a
virus)...

I suppose the beginning of the chicken and egg problem was pretty much
as Nick has already outlined. Primarily, vxers collections often
contain considerable crud, and they rate av scanners on their
indiscriminate ability to detect it all. So some av vendors started
adding detection. And the race was on.

so - for an anti-virus to differentiate between real viruses and crud,
the anti-virus would have to know when it's alerting falsely... and if
it could do that, why would it bother issueing false alerts at all?

Hey, if Frisk can do it, they all can do it Surely, they can't do
it extremely well, but I'm sure they could vastly improve their
reporting and identify crud they know they're purposely detecting as
the crud it is. For example, F-Prot will identify the BSI crud that it
knows about and let you know what it is.


Art
http://www.epix.net/~artnpeg

 

[kurt wismer]

[snip]

And if you claim I'm wrong about something, please be specific. I have
no idea what you believe I'm wrong about.

either you didn't pose the questions exactly the same way in the past
or there were in fact responses or both... i suspect the former...

then there's the issue of you saying you did it here - here is
alt.comp.anti-virus which in the grand scheme of things is a relatively
new group... maybe you did it elsewhere...

It was certainly on alt.comp.virus several years ago. I remember Axel
Pettinger suggested that I push for or do myself some sort of "crud"
evaluation tests of scanners

i googled... i found the thread in question... from my reading of it,
axel considers such testing to be a special kind of false positive
testing - ergo crud constitutes a false positive to him, ergo you got
an answer to the question you said never got answered...

Not really. I was never satisified with the answers/non-answers I was
getting then. I have no idea why you seem to think that a response or
opinion from someone ... even as astute as Axel, would put the
question to rest in my mind.[/QUOTE]

i didn't say it would put the matter to rest in your mind, i said you
got answers to the question you said you never got answers to...

I've never seen any kind of thorouh-going
discussion on this issue.

until now...

But then as now not a "real" false positive

if it's not a false positive then it's a true positive, and if it's a
true positive then it's not crud... we are, after all, dealing with
boolean qualifiers...

[snip]

But what kind of crud?

potentially all kinds...

For example, I recall at the time, some of the
crud in vxer collections I downloaded were object files and BSIs. How
many typical users have such files on their PCs?

this line of reasoning is egocentric and not very scientific... if you
catch yourself using 'typical users' contraints again, smack your hand
and call yourself bad...

Now, some users may
have the results of botched disinfections. Maybe a very small
percentage of false postive test samples could include these. But it's
nonsense to include crud that most users will never realistically
experience.

it's bad enough that crud is context sensitive in relation to the scope
of the test at hand, but now you're verging on talking about
in-the-wild crud...

[snip]

I strongly disagree. You're not being practical or realistic at all.

i said "more practical"... it's still impractical because defining sets
that include/exclude trojans is very hard... however, it's more
practical than if one also had to deal with such ill-defined set
constraints as "files that are found on typical PCs"...

[snip]

that said, crud is not malware, crud is not viral, and crud is not
wormlike - if it were any of these it should be referred to as one of
these instead... crud is garbage, crud is that stuff that scanners
alert on but shouldn't - all false positive tests have used 'crud'....

Oh? Then why do the "super crud detectors" always show quite low and
competitive false alarms rates in the so called "quality independent
tests"?

because false positive tests don't use 'crud' exclusively...


Oh? By "exclusively" you imply that some crud is purposely included??
If so then why do alleged super crud detectors also manage to score
the lowest false positive rates (speaking generally)?

because crud represents a small fraction of the total population of
non-viral/worm/trojan/etc files... by extension, there should only be a
relatively small number of crud files in a false positive test...

You speak of "should" and it appears that you're expressing your
opinion.

well, that's not exactly how it was meant... it's more along the lines
of "if the plane goes fast enough it *should* be able to take off"...

I'm most interested in what is actually done, and for that
I've never gotten a definitive answer.

then you haven't been digging deeply enough... go to the vtc, check out
one of their tests... pay special attention to the section on testing
protocols... i wouldn't necessarily call that a good false positive
test, mind you...

[snip]

Don't ask me. Ask someone who thinks alerting on them as if they are
actually "for real" is something positive, beneficial or true. As I've
said, the problem with many scanners is that they don't identify the
crud for what it is. It's a reporting problem to some extent.

you can't have your cake and eat it too... the only way to report them
properly is to detect them properly - ergo *you* think detecting them
is positive, beneficial, or true... you just don't know you think it yet...

Real false postives in a virus collection? I have no idea what you
have in mind there.

i really can't put it anymore plainly than you already have... i tried
to explain how/why it happens...

Oh? I've certainly seen enough of it. Why do you say I don't
understand it?

because crud is a catch-all group that includes genuine false positives
and it is impossible for a program to distinguish between an infections
and it's own false positives...

I suppose the beginning of the chicken and egg problem was pretty much
as Nick has already outlined. Primarily, vxers collections often
contain considerable crud, and they rate av scanners on their
indiscriminate ability to detect it all. So some av vendors started
adding detection. And the race was on.

and the second half of the chicken and egg problem is that the vxers
were often using an av program when they decided to put the crud in
their collection...

so, did the av detect crud, prompting the addition to a vx collection,
prompting the detection of the crud - or did the av alert falsely,
prompting the addition to a vx collection, prompting the detection of
the crud...

now, unless you'd like to explain how the implementation of crud
detection can violate causality, i think we have to assume the second
scenario...

Hey, if Frisk can do it, they all can do it

f-prot can only do it some of the time... in those cases, it'd be
detecting things that other products false alarmed on - but faced with
crud added due to f-prots own false alarms, f-prot would not be able to
report it as crud...

i don't really care about the pros and cons of crud detection - your
original inquiry was should crud qualify as a false positive... crud is
a context sensitive concept, in the context of a pure virus detection
test crud falls outside the set of true positives and must therefore be
a false positive... however, in a crud detection test, crud would
obviously fall within the set of true positives and would therefore not
be a false positive...

 

[null]

(e-mail address removed) wrote:


until now...

Perhaps. We're just scratching the surface so far.

if it's not a false positive then it's a true positive, and if it's a
true positive then it's not crud... we are, after all, dealing with
boolean qualifiers...

We're dealing with far more that mere boolean qualifiers when we deal
with the real world

this line of reasoning is egocentric and not very scientific... if you
catch yourself using 'typical users' contraints again, smack your hand
and call yourself bad...

No I won't. I'm mainly interested in the situation in the real world
and if you're not then you're the one who should slap his little
pickies.

it's bad enough that crud is context sensitive in relation to the scope
of the test at hand, but now you're verging on talking about
in-the-wild crud...

Yep

you can't have your cake and eat it too... the only way to report them
properly is to detect them properly - ergo *you* think detecting them
is positive, beneficial, or true... you just don't know you think it yet...

Of course I think that detecting them is postitive and benefiical. I
think many scanners do "know" the difference in many cases but they
don't behave as if they do. They report crud as if the file is
actually viable. For that, the vendors should clean up their act, IMO.

because crud is a catch-all group that includes genuine false positives
and it is impossible for a program to distinguish between an infections
and it's own false positives...

You're talking the general case and I'm talking about the many cases
where crud is known to the scanner as such and purposely alerted on as
if it's not crud.

and the second half of the chicken and egg problem is that the vxers
were often using an av program when they decided to put the crud in
their collection...

I suspect that that alerts by scanners were not a criterea for
inclusion in a collection. When scanner A only alerted on 90% of the
files in a vxer collection, the race was on by certain vendors to
improve that percentage by alerting on the crud.

so, did the av detect crud, prompting the addition to a vx collection,
prompting the detection of the crud - or did the av alert falsely,
prompting the addition to a vx collection, prompting the detection of
the crud...

A bit of both, I suppose. There is a "real" crud problem where
scanners don't know the difference of course.

now, unless you'd like to explain how the implementation of crud
detection can violate causality, i think we have to assume the second
scenario...

I think we have to assume both scenarios.

i don't really care about the pros and cons of crud detection - your
original inquiry was should crud qualify as a false positive... crud is
a context sensitive concept, in the context of a pure virus detection
test crud falls outside the set of true positives and must therefore be
a false positive... however, in a crud detection test, crud would
obviously fall within the set of true positives and would therefore not
be a false positive...

It's not that simple, Kurt. The real world never is.


Art
http://www.epix.net/~artnpeg

 

[kurt wismer]

(e-mail address removed) wrote: [snip]

it also seems from the discussion that you also considered them false
positives at that time...

But then as now not a "real" false positive

if it's not a false positive then it's a true positive, and if it's a
true positive then it's not crud... we are, after all, dealing with
boolean qualifiers...

We're dealing with far more that mere boolean qualifiers when we deal
with the real world

over-generalization... not all things in the real world *aren't* black
and white... set membership (aka categorization/classification) is
boolean in the sense that a thing either belongs to set A or it doesn't...

in this case, a object either belongs in the set of positive alerts
(true positive) or it doesn't (false positive)... when there is
question it is not because there's a 3rd possibility, it's because the
set is poorly defined...

No I won't. I'm mainly interested in the situation in the real world
and if you're not then you're the one who should slap his little
pickies.

you can't have it both ways... either it's a good, scientific test or
it's biased towards personal ideologies about real world environments
that differ from one person to the next...

[snip]

Of course I think that detecting them is postitive and benefiical. I
think many scanners do "know" the difference in many cases but they
don't behave as if they do. They report crud as if the file is
actually viable. For that, the vendors should clean up their act, IMO.

i'm not going to argue with you about how terrible the current state of
reporting is... i agree that it is terrible, but this has nothing to do
with whether crud should qualify as a false positive...

You're talking the general case and I'm talking about the many cases
where crud is known to the scanner as such and purposely alerted on as
if it's not crud.

i'm talking about the general case because crud is a general term... if
you have a beef about a certain class of sample, be more specific about
what class it is...

I suspect that that alerts by scanners were not a criterea for
inclusion in a collection.

why not? do the samples pop out of thin air? do the collectors pull
them out of their bums?

a collector puts a sample in his/her collection because someone or
something told them the sample was a virus (and in the case of someone
that person said it was a virus because someone or something told
him/her it was a virus, and so on, and so on)...

i see no reason to think that anti-virus scanners have never been the
'something' in this scenario...

[snip]

A bit of both, I suppose. There is a "real" crud problem where
scanners don't know the difference of course.




I think we have to assume both scenarios.

then you aren't seeing the recursive properties here... if the scanner
was intentionally detecting crud then the crud's detection was added
because it was in someone's crud collection... how did it get there? go
back far enough and you can no longer blame the crud's inclusion in a
collection on crud detection... the crud collection has to precede the
crud detection, and in that case how does it get into the collection...

It's not that simple, Kurt.

i say it it is...

The real world never is.

this is not a valid counter-argument, absolute statements are almost
always false...

 

[null]

(e-mail address removed) wrote:

(e-mail address removed) wrote: [snip]
it also seems from the discussion that you also considered them false
positives at that time...

But then as now not a "real" false positive

if it's not a false positive then it's a true positive, and if it's a
true positive then it's not crud... we are, after all, dealing with
boolean qualifiers...

We're dealing with far more that mere boolean qualifiers when we deal
with the real world

over-generalization... not all things in the real world *aren't* black
and white... set membership (aka categorization/classification) is
boolean in the sense that a thing either belongs to set A or it doesn't...

Ergo some things aren't black and white which what you refuse to deal
with it seems. The subject is loaded with subjectivity and judgement
calls. Nick outlined one he dealt with in this very thread concerning
a virus that was purposely not detected by vendors.

in this case, a object either belongs in the set of positive alerts
(true positive) or it doesn't (false positive)... when there is
question it is not because there's a 3rd possibility, it's because the
set is poorly defined...

Or there are differences of expert opinions.

you can't have it both ways... either it's a good, scientific test or
it's biased towards personal ideologies about real world environments
that differ from one person to the next...

So join the real world and get out of your silly ivory tower

[snip]

Of course I think that detecting them is postitive and benefiical. I
think many scanners do "know" the difference in many cases but they
don't behave as if they do. They report crud as if the file is
actually viable. For that, the vendors should clean up their act, IMO.

i'm not going to argue with you about how terrible the current state of
reporting is... i agree that it is terrible, but this has nothing to do
with whether crud should qualify as a false positive...

Lumping all crud together as you do is a mistake IMO.

i'm talking about the general case because crud is a general term... if
you have a beef about a certain class of sample, be more specific about
what class it is...

Certain classes of sample? Huh? What's that got to do with the fact
that some scanners are too "stupid" to not recognize some crud as crud
while in other cases the scanners "know" it's crud? Besides that you
have legit arguments both ways as to whether or not a infected BSI is
a virus that should be alerted on.

why not? do the samples pop out of thin air? do the collectors pull
them out of their bums?

a collector puts a sample in his/her collection because someone or
something told them the sample was a virus (and in the case of someone
that person said it was a virus because someone or something told
him/her it was a virus, and so on, and so on)...

i see no reason to think that anti-virus scanners have never been the
'something' in this scenario...

It's possible that some super crud detecting scanner was the start of
the race to catch up and detect more and more crud. OTOH, some idiot
vxers may simply have thrown in infected BSIs, harmless object files,
etc. in the belief that such crud _should_ be detected. That's the
irrational real world that you seem to fail to admit exists. And it
seems equally as plausible that such vxer idiocy was at least half of
of it.

[snip]

A bit of both, I suppose. There is a "real" crud problem where
scanners don't know the difference of course.




I think we have to assume both scenarios.

then you aren't seeing the recursive properties here... if the scanner
was intentionally detecting crud then the crud's detection was added
because it was in someone's crud collection... how did it get there? go
back far enough and you can no longer blame the crud's inclusion in a
collection on crud detection... the crud collection has to precede the
crud detection, and in that case how does it get into the collection...

Explained above. Vxer idiocy.

i say it it is...


this is not a valid counter-argument, absolute statements are almost
always false...

Then why do you make so many of them?


Art
http://www.epix.net/~artnpeg