Anti-Malware , Risk Management , Technology

Security products haven't been inspiring a lot of confidence lately. A growing batch of research has shown that security software contains vulnerabilities that can be extremely useful for attackers. A presentation scheduled for Black Hat, which kicks off July 30 in Las Vegas, will reveal a cluster of problems affecting anti-virus applications and other software, including Microsoft Office.

See Also: Rethinking Endpoint Security

A variety of applications do something called "hooking" to get greater insight into another application's behavior. An application's processes and API calls are intercepted, injected with new code and then observed. The invasive process is a critical one for security applications, which need to figure out if something malicious is afoot. Hooking is also used by virtualization, sandboxing and performance monitoring programs.

But researchers at the security firm Ensilo say they've found a half-dozen problems in hooking engines, including Microsoft Detours, an open-source engine called EasyHook, and proprietary engines used by about 20 other vendors, including Trend Micro, Symantec and Kaspersky Lab. The flaws enable attackers to bypass built-in operating system protections, such as Microsoft's ASLR and Control Flow Guard, as well as third-party defenses against exploits.

Any would-be hacker would first need to find a separate way to gain remote access to a targeted system, says Udi Yavo, Ensilo's co-founder and CTO. But once in, an attacker could hack a hooking engine to force the software to inject malicious code into any system process, he says.

Think of it as a force multiplier. For example, a simple buffer overflow vulnerability that gets automatically blocked by security improvements baked into Windows in recent years might once again become exploitable, Yavo says.

ASLR could also become irrelevant because it's possible to figure out the memory addresses of relevant operating system functions.

"Even simple things [become] far easier to exploit," he says. "You don't need to find a place to inject your code. You can simply use places used by these vendors, which are in predictable addresses."

Who's on the Hook?

One of the most popular hooking engines is Microsoft Detours, which Ensilo says is wrapped into the products of about 100 independent software vendors. Microsoft has been notified and is scheduled to patch Detours next month, Ensilo writes. Ensilo suspects the vulnerability has been in Detours since the third version of the software was released, some eight years ago.

The Microsoft Office suite is also affected. Office has a virtualization mechanism called App-V, and Detours is used in App-V, Yavo says. That essentially means that millions of devices are vulnerable. While Office 2010 is not vulnerable, all newer versions are at risk, Yavo says.

Since finding these issues, Ensilo says it's notified a raft of vendors over the past eight months, including AVG, Avast, BitDefender, Citrix, Emsisoft, Webroot, Symantec, Kaspersky Lab and Trend Micro. Those vendors quickly patched the issue. But Yavo says a major anti-exploit vendor and another major anti-virus vendor - both of them have yet to be publicly named - have yet to patch their software.

Ensilo isn't revealing much about how an attacker could use the vulnerabilities, presumably because not all vendors have patched, as well as to save some punch for its Black Hat presentation. But the company says that patching isn't straightforward, because each vendor must recompile affected applications.

Also, Yavo says software vendors are dependent on the hooking engine developer to first deploy a patch before they can recompile their applications.

Seeking Secure Security Software

The hooking problem adds to growing concerns that many anti-virus products might be riddled with dangerous, unknown vulnerabilities that could completely undermine a system's security.

Last month, Google Project Zero researcher Tavis Ormandy - for the second time - found a vulnerability in Symantec's anti-virus engine that could be exploited merely by sending someone an email (see Second Symantec Anti-Virus Bugfest Found). Ormandy's finding added to a long list of problems he's found in products from such security software vendors as Avira, ESET, FireEye, Kaspersky Lab and Sophos.

Meanwhile, expect more details on hooking engine hacks via Ensilo's upcoming "Captain Hook: Pirating AVs to Bypass Exploit Mitigations" Black Hat presentation, with Yavo and Tomer Bitton, Ensilo's vice president of research. It is scheduled to take place on Wednesday, Aug. 3, at 4:20 p.m. in the Jasmine Ballroom at the Mandalay Bay in Las Vegas.