There is a good reason we appear to have "missed" some of the items in that test. The most thorough explanation for this is provided on our Threat Blog in an explanation co-authored by both Joe Jaroch, our Vice President of Endpoint Solutions Engineering, and Grayson Milbourne, our Director of Threat Research. It's a fascinating and informative article.
The short version below is provided by Mike Malloy, our Executive Vice President of Products and Strategy:
In summary, what happens with this test is this:
- Samples are installed on a PC with WSA. We immediately and correctly detected 93% of the malware samples upon initial exposure.
- Under the procedures of the test, once the AV product has either detected the sample as bad or not, AV Comparatives immediately re-images the machine to conduct the next test on a fresh sample, and, if we do not indicate that we had blocked it, we are recorded as having missed the sample.
- However, this test ignores how WSA actually functions. If we are unable to make an immediate determination, we do the following:[list=1]
- Start journaling all the activity of the new sample.
- Mark the sample for re-determination.
- Begin frequent, every minute, checking with the Webroot Intelligence Network to see if the sample has been determined.
- Once the determination comes in, often within minutes, we either quarantine and rollback all the activity the sample may have done (if it’s bad); or we take the marker off the sample and stop the journaling (if it’s good.)
But the AV Comparatives test aborts this crucial process when it re-images the test PC.
In fact we determined all but 3 of the more than 600 samples within two hours of each test. During the time between 1st exposure and determination, our customers would still be protected by the combination of the journaling/rollback and the identity shield feature which prevents personal information from being hijacked.
We highly respect AVC and support their independent testing. We are working with AVC to develop a test that both reflects real world use and reflects the true efficacy of our product.
