Botnets Could Meet Their Match in Robot Hackers

Last summer the Pentagon staged a contest in Las Vegas in which high-powered computers spent 12 hours trying to hack one another in pursuit of a $2 million purse. Now Mayhem, the software that won, is beginning to put its hacking skills to work in the real world.

Mayhem was created by security startup ForAllSecure, cofounded by Carnegie Mellon professor David Brumley and two of his PhD students. Brumley says the company has started adapting Mayhem to be able to automatically find and patch flaws in certain kinds of commercial software, including that of Internet devices such as routers.

Tests are underway with undisclosed partners, including an Internet device manufacturer, to see if Mayhem can help companies identify and fix vulnerabilities in their products more quickly and comprehensively. The focus is on addressing the challenge of companies needing to devote considerable resources to supporting years of past products with security updates. Late last year, hackers used a massive botnet of compromised Internet devices such as cameras to take down sites including Reddit and Twitter.

“Now when a machine is compromised it takes days or weeks for someone to notice and then days or weeks—or never—until a patch is put out,” says Brumley. “Imagine a world where the first time a hacker exploits a vulnerability he can only exploit one machine and then it’s patched.”

Last year, Brumley published results from feeding almost 2,000 router firmware images through some of the techniques that powered Mayhem. Over 40 percent, representing 89 different products, had at least one vulnerability. The software found 14 previously undiscovered vulnerabilities affecting 69 different software builds. ForAllSecure is also working with the Department of Defense on ideas for how to put Mayhem to real world use finding and fixing vulnerabilities.

The Cyber Grand Challenge contest Mayhem won last year was staged by the Pentagon’s Defense Advanced Research Projects Agency, DARPA, in an attempt to spur research on the idea of automating some of the work of security experts. Teams…