How not to measure PC security tools

Graham Cluley
Graham Cluley
@[email protected]

There’s an interesting article by Mark Ward on the BBC News website today, where security firm Prevx (you’ll remember them from their part in the BBC Click botnet fiasco) proposes a new way of measuring the effectiveness of anti-virus software.

According to the report, Prevx will..

..create and distribute a small program that will gather statistics on how quickly security companies find and remove malicious code. The figures will reveal if users are being left vulnerable and for how long as well as rank response times.

Apparently the proposed tool will “log when files are installed.. [and] would alert a user if it noticed that a fix had been created for a particular virus or trojan it had spotted on a PC.”

Sign up to our free newsletter.
Security news, advice, and tips.

BBC News story

Leaving aside for now the suggestion in the report that anti-virus companies have to react to every new malware threat (what about our proactive detection?), there are a number of problems with this method of measuring the effectiveness of security software.

Off the top of my head, and in no particular order:

1. Privacy. Are people who install the tool going to be happy with a third-party application keeping a record of when every program is installed? As malware can be spread in a wide variety of file formats, this means logs will also need to be kept about Word documents, Excel spreadsheets, PDF files and much more. Will these statistics be kept purely on the PC or communicated via the internet? Will people have legitimate concerns about what might be done with that data?

Quarantine2. From the sound of things the tool will only attempt to measure when an infection occurs and is fixed later by an anti-virus update. What if the infection was prevented in the first place?

For instance, if the users’ web or email gateway intercepted the malware as it tried to enter the organisation via an infected website or email and has blocked the infection or quarantined it before it managed to hit your computer?

In those cases, nothing on the users’ hard drive will have changed, and Prevx’s logging software won’t have anything to log as no malware was ever installed. So much for measuring effectiveness.

3. How will the tool confirm that an infection has been properly cleaned-up? Prevx says that their tool will be a “small program” (which is a good thing), but will it be able to confirm itself that an infection has been properly removed or will it rely on messages from the user’s anti-virus software to confirm that?

And if the latter, how is that measuring true effectiveness? What if the anti-virus software mistakenly says it has cleaned up an infection but has done a poor job – how will that be measured?

4. Why would people install a tool like this? What’s in it for them?

Yes, it would be great to have reliable information on the success of anti-virus software – but I can’t see what the incentive is for the typical user who is trying to eke out every last cycle of their CPU to run this type of software. It isn’t clear what benefit it brings users.

Install5. Why would companies install a tool like this?

In my experience, most system administrators want tight control over the software that runs on their networks. I can’t imagine a tool like this is going to win much favour with most of them as they try and maintain uptime and ensure that incompatibilities and program clashes are kept to a minimum.

6. How will the tool know how the infection was cleaned-up? We’ve already ascertained that the tool shouldn’t rely on the very anti-virus product it is trying to test to confirm if it has succeeded in cleaning-up an infection or not.

So imagine the scenario where Aunty Hilda’s PC gets infected by the Plastic Pizza Trojan. She tries the SuperScan Anti-Virus program and it fails to get rid of the infection. Then she downloads a trial edition of MightyAntiMalware’s security product and it sorts her out. How will Prevx’s tool know which product disinfected the PC?

7. Who’s going to pay for the infrastructure behind this tool? Presumably it’s intended to collect data from millions of computers around the world – data which should be kept securely. Furthermore, it may require frequent updates in an attempt to handle some of the issues I raised above. And yet, I imagine it’s going to be given away for free.

8. What happens when the free tool gets it wrong? What if anti-virus companies swear blind that they had been detecting the Plastic Pizza Trojan for a good three weeks already, and that Prevx’s tool is incorrect to say that they left users unprotected? Or if Prevx’s tool swears blind that a user was infected when they weren’t?

With traditional tests it’s possible to go back and re-test to see if a mistakes has occurred – but that’s going to be nigh on impossible with this tool.

Tape measure
I think we all recognise that there is room for improvement in security products, and we need better ways to measure the effectiveness of the different solutions in the marketplace. Groups like the Anti Malware Testing Standards Organisation (AMTSO) are working hard to build standards for testers, which should make things better for purchasers of products.

Prevx’s tool may have noble aims – but to my mind, the concept of a free tool that measures the effectiveness of your anti-virus product on your PC is half-baked until the above concerns are adequately dealt with.

Image source: Darren Hester’s Flickr photostream. (Creative Commons)

Graham Cluley is an award-winning keynote speaker who has given presentations around the world about cybersecurity, hackers, and online privacy. A veteran of the computer security industry since the early 1990s, he wrote the first ever version of Dr Solomon's Anti-Virus Toolkit for Windows, makes regular media appearances, and is the co-host of the popular "Smashing Security" podcast. Follow him on Twitter, Mastodon, Threads, Bluesky, or drop him an email.

What do you think? Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.