Yahoo CISO Details Challenge of Security at Scale

LAS VEGAS: Since joining Yahoo as its CISO six months ago, Alex Stamos has been focused on one big challenge: how to do security at scale. In a standing-room-only session at the Black Hat USA conference, Stamos detailed why security products don’t work for Web-scale companies and explained what Yahoo is doing to improve its security posture.

“We have failed to make the Internet safe because we haven’t thought enough about scale,” Stamos said.

In the post-Snowden era, many people are just chasing the phantoms of incredibly advanced threat actors, Stamos said. That’s the wrong approach, he believes, since the vast majority of real users are not targeted by nation states or advanced threat actors.

Talking about Yahoo and Web-scale companies specifically, Stamos argues that many security vendors don’t offer solutions that work. The problem in his view is one of focus.

Focus on Financial Services

Stamos said that security companies sell lots of product to banks and tend to build their solutions to match the needs of these financial clients. Banks are not Web-scale companies, however, so when security vendors pitch their gear to Stamos, he’s somewhat less than impressed.

The biggest banks have tens of millions of customers, while Web-scale companies like Yahoo have hundreds of millions. Banks have hundreds of front-end servers, while in the Web scale servers are measured in the tens of thousands. Vendors try and sell him server solutions that match up one-to-one to protect Yahoo servers, he added, noting that this approach cannot scale.

For Yahoo, what makes more sense is a system of software sensors that combine with centralized intelligence. Some vendors have tried to sell him centralized intelligence, Stamos said, but with a non-distributed back-end architecture.

“People try and sell us database backed SIEM (security information and event management) systems,” Stamos said. “At Yahoo we have lots of petabytes of storage, so it’s not cost effective for us when all data is dumped into one database.”

What Stamos needs is anomaly detection that works on dumb storage clusters that run on cheap hardware.

“That’s why we invented Hadoop and use it everywhere,” Stamos said. “We have to build security products that can spread across tens of thousands of machines.”

Bug Bounty and Patching

Yahoo also faces a challenge of scale in dealing with bug reports. Yahoo has a robust bug bounty program that awards researchers for flaw disclosures. So far in 2014, Yahoo has paid out $670,000 to researchers as part of the bug bounty program, Stamos said.

Stamos wants to scale up the bug bounty program, though, so it can be even faster and more efficient. He is planning a bug bounty system with automatic verification using the open source Selenium project. The system will enable a researcher to file a bug in the system, which will be automatically tested. If the bug is deemed to be a priority, a bug file can be sent to a Yahoo engineer for further validation and remediation.

Making sure that systems are patched quickly is another challenge of scale. “If a machine is not important enough to be patched, it’s not important enough to still be running,” Stamos said.

Sean Michael Kerner is a senior editor at eSecurityPlanet and Follow him on Twitter @TechJournalist.

Sean Michael Kerner
Sean Michael Kerner is an Internet consultant, strategist, and contributor to several leading IT business web sites.

Top Products

Related articles