So which endpoint protection product should your organization purchase? It's a difficult question to answer amidst shouts from overconfident vendors and swirling FUD. Industry anti-virus tests would be the easy answer, but unfortunately, not all vendors participate.
See Also: 12 Top Cloud Threats of 2016
To be sure, anti-virus vendor scuffles have stepped up a notch lately, with startups and industry stalwarts slinging mud more fiercely than ever. But one of the newer entrants, Cylance, says it plans to make its Protect software available for industrywide tests used to benchmark software, which industry experts welcome.
Cylance is one of several so-called next generation AV vendors that say they rely on a combination of machine learning and algorithms to detect malicious behavior. Many security software applications still rely on signatures - frequently updated descriptions of known harmful code - to detect malware.
Signatures are a good way to detect malware, but are unfortunately created after a piece of malicious code has been used in an attack. Signatures can also miss even known malware if it has been modified or compressed, making them an unreliable backstop.
The new approaches by next-gen AV vendors mean that some malicious code may not be detected right away. Instead, the products look at behavioral aspects of a new piece of code that might indicate it is harmful and should be stopped.
To Test, or Not to Test
As a result, next-gen vendors may not appear to be as effective in a simplistic test of one product vs. another against a set of malware samples. That has caused newer vendors to shy away from testing, fearing that a poor result on a simplistic test will cast a pall over their products - and potentially cause venture capital funding to dry up.
Chap Skipper, vice president of certification and testing for Cylance, says that testing needs to center on what defense contractor Lockheed Martin has popularized as the "cyber kill chain." It comprises seven stages of a cyberattack, all of which offer opportunities for detection.
Skipper joined Cylance last month after working in Dell's CTO office. He was instrumental in Dell securing a partnership with Cylance after Dell sought a security industry partner. He says it's important the industry agrees on how something is determined to be malicious, or "convicted," because next-gen products can make that determination in varying ways.
Even if a computer gets infected, a security product may halt the virus after it does other actions later, such as contacting a remote server, creating new processes or installing other obfuscated code.
"There's no silver bullet in security, and we're human," Skipper says. "I've seen Cylance miss [malware] even in my own testing. But it's now about engaging those third-party testing organizations to see if we can't have more of a cyber kill chain type method and understand their conviction process."
Tested Anyway
AV-Comparatives, a testing organization based in Austria, and MRG Effitas, a U.K. based research organization, published results in February comparing Cylance's Protect product and Symantec's Endpoint Protection. The report noted the companies had trouble obtaining Protect from resellers but eventually secured a copy from an unnamed third party.
"This behavior is seen by many of the newer products that claim to be next generation," writes AV Comparatives. "It looks like they try to avoid getting tested in order to continue to attract users simply by unproven marketing claims."
Since that test, AV-Comparatives has been in touch with Skipper.
Testing a product without permission is a thorny issue, and many security companies forbid it in their terms and conditions. A Cylance competitor, Sophos, obtained a copy of Protect and published a YouTube video of its own product test (see Anti-Virus Wars: Sophos vs. Cylance).
Sophos, perhaps unsurprisingly, came out on top. Cylance was furious and pressured the reseller who gave its product to Sophos. The video was taken offline, although Sophos says it was to take the heat off of the reseller.
Down With the Kill Chain
AV testing organizations say they're open to new ways to conduct tests and have already moved in that direction.
Maik Morgenstern, CTO for AV-Test, says that his organization already does full cyber kill chain tests, including from the original infection to execution of code. The test takes into account some of the many defenses built into security products, including URL blocking, static scanning, dynamic detection, reputation-based blocking and cloud detection.
"We are confident that our testing methodology is fully applicable to next-gen products as well," Morgenstern says. "As we are testing with the real threats and do so as a user would do (visit websites, receive emails) we are covering exactly that."
Simon Edwards, founder of the testing organization SE Labs in London, says just testing for an intrusion isn't enough. Otherwise, the test results aren't very useful for buyers who are trying to figure out which is the best product.
"Testing many modern security products requires that the tester expose them to the full range of attack elements," Edwards says. "Testers can't make assumptions about how these products work and what they do."
Time will tell how much Cylance and other vendors decide to open up. But if they do become more open, end users will benefit, and, hopefully, enterprises won't have to make shot-in-the-dark decisions when buying security software.