About 90% of security buyers aren’t getting the efficacy from their products that vendors claim they can deliver, according to a panel at this week’s RSA Conference that also offered some solutions to begin to fix that problem.
Joe Hubback of cyber risk management startup ISTARI led both the panel and the study, which was based on in-depth interviews with more than a hundred high-level security officials, including CISOs, CIOs, CEOs, security and tech vendors, evaluation organizations and government organizations.
Hubback said that “90% of the people that I spoke to said that the security technologies they were buying from the market are just not delivering the effect that the vendors claim they can deliver. … Quite a shocking proportion of people are suffering from technology that doesn’t deliver.”
A number of reasons for that product failure came out in the panel discussion, but they can be boiled down to some key points:
- Cybersecurity buyers are pressed for time and most don’t test the products they buy. “They’re basically just buying and hoping that the solutions they’re buying are really going to work,” Hubback said.
- Vendors are under pressure from investors to get products to market quickly and from sales and marketing teams to make aggressive claims. On top of those pressures, it’s difficult to architect tools that are effective for a range of complex environments – and equally difficult for buyers to properly assess these “black box” solutions.
Also read: Latest MITRE Endpoint Security Results Show Some Familiar Names on Top
A ‘Market for Lemons’
Those dynamics create an “information asymmetry,” where vendors know their product and its strengths and weaknesses, but buyers don’t have the time or information to understand all their options.
“This information asymmetry is the classic market for lemons, as described by George Akerlof in 1970,” said Hubback. “A vendor knows a lot more about the quality of the product than the buyer so the vendor is not incentivized to bring high-quality products to market because buyers can’t properly evaluate what they’re buying.”
Panelist Greg Rattway, a former JP Morgan Chase managing director and co-founder of cyber risk management firm Next Peak, took issue with the term “lemons.”
“Vendors are not trying to develop lemons,” Rattway said, “but the challenge is trying to develop a tool that will do the things that marketing says you have to market aggressively, and therefore the claims for the tools and the technologies are so dramatic, but the complexity of the environment almost makes it impossible for development of a single technology or tool that will actually solve problems in a complex environment.”
Grace Cassy, co-founder of the CyLon cybersecurity startup accelerator, and a number of others agreed that the complexity of IT environments makes the challenge for product development teams – and buyers – even greater.
“It’s hard to generalize from a test environment,” Cassy said. “What works in a test environment might not in another environment.”
One audience member said a network of peers can be helpful forbuyers – but his network discusses use cases a product is effective for rather than generalizing product opinions.
And with marketing and sales budgets far outweighing product development budgets, some said vendors might even welcome pressure from buyers to be less focused on financial results and more focused on product development.
Also read: Testing & Evaluating SIEM Systems: A Review of Rapid7 InsightIDR
Buyers Need to Improve Too
Two-thirds of the security pros interviewed by Hubback said cybersecurity buyers need to do better too. Part of that is organizing and placing demands on vendors.
To that effect, Hubback, Cassy, Rattway, and University of Oxford government professor Ciaran Martin – the founding head of the UK National Cyber Security Centre – discussed efforts to develop the Buyer’s Charter for SAFER cybersecurity, dubbed “a vital next step to address the market failings.”
SAFER stands for:
- Symmetry: Fixing the “symmetry of information” problem
- Assessments: Developing transparent assessments
- Freedom that allows for maintaining innovation and new market entrants
- Efficacy: Developing a framework for defining efficacy
- Risk-based decision-making: Bringing all those aspects together to allow for better decisions
Hubback said the group is essentially proposing something similar to GSMA’s role in setting telecom standards, “a demonstration of ability.” He invited RSA attendees to join the group’s efforts.
See the Top Cybersecurity Companies for 2022
Defining Cybersecurity Product Efficacy
Hubback said cybersecurity product efficacy depends on four things: capability, practicality, quality and provenance.
- Capability: Does this technology have the capability to deliver what was promised and can the effectiveness be demonstrated?
- Practicality: As Hubback put it, “Is this actually something I can use in my company without needing some kind of Top Gun pilot to fly it and make it work?”
- Quality: How well designed and built is the solution to avoid vulnerabilities and negative impact?
- Provenance: How much security risk is in the vendor and its supply chain, based on how they work and who they are?
“Now clearly there are some efficacy measures like, is it in the Magic Quadrant, or do all the other CISOs who I text say that this works?” Hubback said.
But he noted: “As an industry, it’s quite interesting that we don’t really have a structured way of doing it today, so this is one of the things we’re sharing from this research is a proposed view of how to define the efficacy of security technologies.”
Read next: Top Endpoint Detection & Response (EDR) Solutions
