Ruh-roh.
AV-Comparatives, one of the world’s leading independent testers of anti-virus products, says that it has uncovered that at least one product isn’t playing by the rules.
AV-C has uncovered an infringement of the testing agreement by one of the vendors participating in its tests. It has been found that a product submitted for testing by the vendor had been specifically engineered for the major testing labs, including AV-C; public availability of this version was limited. A second vendor is also being investigated for similar reasons. When this analysis is complete, AV-C will announce the measures it will take against the vendor(s) found to be in breach of contract.
Imagine if the security software you or I might get from an anti-virus vendor was different from what AV-Comparatives tested. Frankly, what would be the point of reading the test at all?
It would clearly be a problem if a vendor was able to ensure that the version of its software tested by AV-Comparatives was “tweaked” to distort its capabilities in the real world.
Right now, mystery surrounds who the offending vendor might be or the details of what they have done. But it’s possible that there is more than one offender.
AV-Comparatives says it suspects another vendor may also be guilty of breaching the rules of the tests, and that it has informed other testing bodies of what it has found so far.
A posting on AV-Comparatives’ Facebook page reveals little more in the way of new clues as to who the offending security vendors might be.
In the next few days we will probably know more, but right now one thing is for sure. All security vendors will be on their guard to make sure that they’re not found trying to cheat the results of anti-virus tests.
Update: Revealed: The anti-virus vendor cheating in independent tests
Well… The AV producers will always try to get more favorable tests. Submitting a non-standard version to the test is rather crude. There are much more elegant solutions.
If I remember correctly, Dr. Solomon had introduced the following "feature" in his scanner. By default (i.e., unless a special command-line option was used), the scanner would recognize a situation when it was detecting more than 5 different viruses. Since this is unrealistic to happen in real life, it would decide that it was being run in a test (i.e., on a virus collection) and would drop its exact identification algorithms, in order to speed up the scanning.
This was done because Alan was fed up with testers inappropriately testing the speed of his scanner on an infected machine (instead of on a clean one). But it is easy to imagine a more nefarious use of this idea. If the scanner detects that it is being run in a testing environment, it could simply start flagging all samples as "possibly infected", in order to score a better detection rate in the test.
One way to circumvent that is to mix the virus collection with clean files and penalize the scanner for false positives – but how many testers are going to bother? It's much easier to run the detection test and the false positive test separately, on two different sets of samples.
Hey Graham, I was just reading the 2014 complete AV test for mobile products. I downloaded the pdf. There were problems with Qihoo and I think Baidu,or some other Chinese vender. How much you want to bet that the offenders are Chinese? If you want to save time serching,let me know and I will send you the pdf file.
Now,the guy above mentions testing ,and I can tell you that AV-C does mixed testing,so as to make sure to catch venders who make false positives. They are very comprehensive in their testing. AV test .com I think only feeds malware samples and false samples seperately ,I think,I could be wrong. I like AV Comparative better,because they test every feature of the apps and software. Much more informative and thorough.IMHO
Well… good call on Qihoo because indeed that is the first one to suffer the consequences of cheating. Sad really and very unhelpful. Worse than that it is harmful because malware is such a problem to the end users as well as servers. That isn't even considering things like DoS[1]/DDoS attacks, randomware and other similar things but those are obviously also quite serious (to the end user especially).
[1] Including any instability of the host (to use a more technical term even though it isn't authorised) of the malware.
Wouldn't it be better if AV-Comparatives went out & bought retail versions of their test candidates?
Lee that's an idea, but it won't work. All the av manufacturer has to do is include a small routine in their scanner to test for laboratory conditions. Once those conditions are met, for example when a scanner detects over 5 unique viruses in a single sitting, it switches into enhanced mode. The test-for-lab-condition routine is in the av software regardless if it comes from private or public distribution.
Vesselin you are correct about Dr Solomon's Software. This article reminded me of that period in history.
Hi Graham,
Apparently I was right about my first speculation. Here is the link to the register article.
http://www.theregister.co.uk/2015/05/01/cheater_test_labs_out_av_vendor_for_using_rivals_engine/
Chinese anti-virus vendor Qihoo 360 has been caught cheating on benchmarking tests by submitting versions running A-V engines from rival Bitdefender.
The company has been reprimanded by established testing outfits VirusTotal, Av-Comparatives, and AV-Test which withdrew its 2015 certifications.
In a joint statement [PDF] the AV testing outfits say Qihoo "cheated" by turning off its engine and flicking on BitDefender's, a setting state that is the opposite of what customers receive by default.
Maybe they refer to Yet Another Cleaner. They were exposed to have illegally using Malwarebytes' database. See here: https://blog.malwarebytes.org/fraud-scam/2015/03/yet-another-cleaner-yet-another-stealer/