In cybersecurity, there are too many variables on both the attack and defense sides to easily calculate ROI for specific spends

You cannot separate cost and value in business: value is used to justify cost. Business value is measured by the return on investment (ROI) from cost. By understanding current ROI it is easier to justify future cost because you know the value. But this is a problem: how do you measure or quantify ROI in cybersecurity spend?

The problem

“A good day in security is when nothing bad happens,” says Sounil Yu, CISO at JupiterOne. The problem for understanding ROI is why did nothing bad happen? Was it luck, and on that day, you were not attacked by an elite hacker? Was it because you maintain a thorough patching program? Was it because of one or more of your cybersecurity controls – but which one or ones were successful, and how much cost to the firm did they prevent? None of these is easy to explain or quantify if nothing bad happened.

But, continues Yu, “Calculating some form of value from security expenditures becomes necessary for security leaders to differentiate luck from skill.” This is important information to convey to the board or whoever controls the security budget. ‘Luck’ can promote an optimism bias; that is, the belief that since nothing has happened, nothing is likely to happen. This will make it harder to obtain future budget because it may be considered unnecessary.

Yet, understanding what security controls have been effective (to convey the skill level) is difficult. “There is limited data to make reliable estimates on likelihood,” said Yu. “For example, the massive increases in cyber insurance this year resulting from waves of successful ransomware attacks represents the gross miscalculations of likelihood made by most insurers. In other words, those who are highly incentivized to use rigorous actuarial methods to calculate the value of security controls still got it quite wrong.”

Nevertheless, comments John Hellickson, field CISO at cybersecurity firm Coalfire, with increased board oversight of cybersecurity, “It’s important to tie specific cyber investments that can show improvement to cyber maturity and reduction of risk to key business objectives.”

The problem in cybersecurity is that there are too many variables on both the attack and defense sides to easily calculate ROI for specific spends.

Some areas can be quantified

Not all elements of an ROI calculation are impossible – for example, the cost of specific failure in certain areas. “There are areas where you can quantify losses, making the cost of mitigating controls realistic,” says Rick Holland, CISO and VP of strategy at Digital Shadows. He cites the cost of lost revenue if an ecommerce site is forced offline (which can be used to justify DDoS mitigation spend); while B2C companies can forecast the impact of stolen credentials (justifying spend on enhanced authentication solutions).

Taking this approach to its logical conclusion, the CISO can approach the board with a total cost of cybersecurity failure and a budget request to mitigate all loss. It’s a nice idea, but one that won’t float. The board will not entertain total failure, but will demand to know the likelihood of individual failures.

“Putting a percentage likelihood number on the probability you will be breached is very subjective, and I’d be skeptical of most organizations’ ability to do this,” says Holland. “When quantifying risk in economic terms, there are so many variables that are challenging to calculate,” he adds. “There is no ‘easy button’ when quantifying cybersecurity ROI; for most companies, it can be more art than science.”

This is an important comment, because it specifies the two primary but opposite approaches: cybersecurity as an art and cybersecurity as a science.

Treating cybersecurity as an art

Bernard Montel, technical director EMEA at Tenable, remembers the time he was asked how he would recruit engineers for a SOC. “The answer was, I don’t want to have an expert on firewalls or pentesting. I would love to get a gamer – someone who never gives up, someone with a lot of curiosity, someone who wants to discover maps or some part of the game they’ve never seen before and try again and again and again. That is better mindset for me. Someone doing, you know, hunting or investigations rather than just having a subject expert on network security.”

This use of personal experience, knowledge and understanding and being able to think outside the (scientific) box is a good example of the art of cybersecurity.

Jadee Hanson, CIO and CISO at Code42 is a firm believer that successfully implementing security is an art form. She’s not even keen on the term ‘ROI’, preferring to call it cost/benefit analysis. The key areas are understanding your company’s security maturity level, understanding the company’s risk acceptance levels, and making what is essentially a subjective decision on the areas that need to and can be maintained or improved.

She thinks of security as an internal insurance policy to protect the ROI of other parts of the business. “At the end of the day,” she said, “security is a G&A (general and administrative expense) function of the organization. We function to protect the ROI for other parts of the organization that generate true revenue.”

Marketing is an example. “Let’s say marketing has a target RoI of 10% more revenue resulting from marketing spend. In security, our task is to have the right security control, the right deployment and the right configuration of that product to protect marketing’s ROI by protecting the technology used by marketing.”

The way to achieve this is through a thorough understanding of the business and its goals, which is achieved by balancing the company’s security maturity against the company’s risk tolerance. The former is controlled by available budget, while the latter will vary from firm to firm.

“If you’re a smaller company, you can afford to take on a lot more risk. Your culture is one that is already centered around risk taking; so, you’re going to have a lower budget and you’re only going to focus on the most important items. If you’re a larger company, or regulated, your culture is one where you can’t afford any security misstep. You’ll have a higher budget and you’re going to focus on closing as many risks as possible via people, process and technology.”

Missing from this argument is stressing over security spend ROI. The key is understanding the business expectations rather than the science of probability, and then aligning risk tolerance (which is a variable) with actual risks (which vary) in accordance with available budget (another variable) and available controls. The available controls are the biggest variable. Even if you can find a product that promises what you need, and has performed for other companies, it will only work until it doesn’t. And that is something science cannot predict.

Gaining budget is an art, because it is heavily dependent on the CISO’s presentation of requirements. Using budget wisely is also an art, because it depends on the CISO’s personal knowledge of an ever -changing threat and product landscape, personal relationships with peers for information sharing, and personal relationships with vendors to get the best deal possible. And still the mitigation only works until it doesn’t.

Science

Stan Black, CISO at Delinea, leans toward the scientific approach. “Of the (primary) types of risk treatment [avoidance, reduction, transfer, acceptance],” he said, “cybersecurity ROIs generally fall into two main categories, risk avoidance and reduction. Both categories can be quantified in ratios of cost versus financial risk. For an example, if we implement privileged access, the risk of privacy fines and legal fees will be reduced by nn%.”

Richard Seiersen, the CRO at cyber insurance firm Resilience, is a strong believer in the scientific approach to ROI quantification. “My job,” he told SecurityWeek, “is to build quantitative models for insurance, working with our actuarial science and data science team.” He has a background in quantitative science, being the author of a standard textbook (How to Measure Anything in Cybersecurity Risk), and more recently, The Metrics Manifesto.

His basic view is that although actuarial data for cybersecurity is more limited than other insurance areas, the science of probability is designed to produce accurate forecasts from limited data. “Is it precise? No. Is it accurate? Yes.”

He used ransomware as an example. “We have a lot of data on extortion,” he said; pointing out that even the criminals use a form of ROI forecasting while setting their extortion fees. “You don’t see extortion fees that are beyond the revenue of the victim.”

The amount of data available from ransomware attacks is continually growing. “We have extortion and then we have business interruption. So, we start correlating the details and get into the math and can begin to do some forecasting. The question becomes, what’s the buy across my whole portfolio based on the cost of control relative to its value in reducing the likelihood of loss? Which set of controls have the best return on investment from a dollar perspective. What is the cost of the controls that will give me the best reduction of probable future loss?”

Seiersen believes all CISOs already do this in at least an informal manner even where they reject the scientific approach. “They’re doing what I call naïve benchmarking. What does Gartner say? What does Forrester say? I’ll get on Slack and see what my CISO peers believe. I’ll ask what they think about this control versus that control. They’re doing a vague benchmark, looking at cost relative to the priorities – and then they’re placing a bet.”

This, he suggests, is normal and what most people do all the time. “But it’s a hyper naive, semi quantitative approach to doing things. I’m suggesting it can be done better.”

He is a fierce believer that the formal, scientific approach can lead to a better understanding of both existing and potential ROI on security spend. “Probability is a tool used to measure subjective forecasts. That’s what it is used for. Anyone rejecting this is standing against the whole history of science, and that doesn’t make any sense to me.”

Is it even necessary?

There is one question left unasked in this art versus science approach to calculating ROI. Is ROI even necessary? Are we too hung up on the concept of return on investment in cybersecurity spend? Hanson believes we probably are.

“Security’s function is to protect the ROI of the business departments that actually generate revenue for the business. As a G&A function, it is more like HR or legal than marketing or sales or manufacturing. I think we must move away from thinking of it as part of the organization that increases revenue and think of it more as just a standard function that every organization should have in place.”

“I don’t think there is a single strong answer,” suggests Chris Morales, CISO at Netenrich. “It really comes down to the risk appetite of the organization and what they are trying to achieve. What is the risk and is that risk worth taking? Controls and actions should be less than the potential loss but done right it should enable the business to grow and succeed.”

By Kevin Townsend on Tue, 20 Sep 2022 13:48:57 +0000
Original link