The great and the good of the security world returned from Budapest last week where there was a meeting of AMTSO, the Anti-Malware Standards Organisation.
One of the outcomes of that meeting was that AMTSO has decided to provide a public analysis of the testing methodology used in anti-virus product reviews. That means next time a consumer picks up a copy of XYZ magazine and reads an anti-virus test, they’ll also be able to go somewhere independent to see how accurate and reliable that test is in the opinion of experts.
Why’s this important? Well, in the past there have been a lot of poor quality reviews done of anti-virus software. That’s not always because the reviewer is deliberately making mistakes, but because testing security software is notoriously hard – and it’s easy to make an elementary blunder and come to the wrong conclusions.
AMTSO’s aim is to improve the quality of testing. As they explain more eloquently than I could:
"Good testing helps vendors to raise the quality of their products, and that's good for the consumer and therefore for sales. Inadequate and unfair testing can mislead consumers by promoting poor products at the expense of better products, and that's bad for everyone, even those products that gain an unmerited marketing advantage." – Source: “Why would I trust an industry body like AMTSO to design fair tests?” – AMTSO website.
It’s important to note that there isn’t a conspiracy here, and AMTSO isn’t run by a particular anti-virus company. All the major anti-malware companies (and quite a few you’ve probably never heard of), as well as respected independent testing bodies, participate in AMTSO. But don’t take my word for it, you can view what appears to be a current list of members on their website.
On the SophosLabs blog, Stuart described that he believed this was “the start of AMTSO influencing the quality of testing for the good of the consumers. AMTSO is not seeking to tell testers how to conduct tests but to inform and highlight what needs to be considered to produce a well rounded test.”
You can read more about AMTSO’s plans in the press release on their website.
I, for one, am really interested to see what comes out of this and am hopefully that the whole security community (including testers, vendors and users) will be served by experts coming together with this initiative.
