Supply chain security and technical debt: Is it weighing your team down?

supply-chain-security-technical-debt

Rampant lapses in software supply chain security don't manifest themselves all of a sudden. They build up over months and years — one out-of-date component, overly permissive account, or misconfigured API at a time. And so, these gaps start mounting up like bad credit card debt on the ledger of supply chain security.

Each individual charge of security technical debt may seem inconsequential, often left behind unintentionally. Many times incurred on purpose — 'for now' — as the DevOps team figures they'll deal with each issue at some undetermined future date in order to speed up the next minimum viable product release or high-priority feature request. Of course, however, most of the time they never get back to those security problems.

Andrew Barratt, vice president at the cybersecurity advisory service Coalfire, said it is arguable that almost all application security lapses come down to a technical debt you were either aware of – "or didn’t know you didn’t know."

"[Supply chain security technical debt] usually stems from a complete lack of refactoring in a highly agile environment."
Andrew Barratt

The problem, of course, is not in each individual security debt. It's in the accumulation. Just like with a high-interest credit card, there's a compounding factor to security technical debt as it's racked up. Compounding interest comes in the form of software supply chain risks.

The potential impact of vulnerabilities grows more serious when each one is combined with other flaws in the supply chain and likelihood of exploitation increases as the volume of flawed components mount across a supply chain. And the more that organizations rack up that security technical debt, the harder it becomes just to pay down the interest—let alone the principal.

If organizations want to get a handle on software supply chain risks, they need to understand how this security debt grows and what it will take to start paying it off the smart way before they face a 'bankruptcy' situation — be it regulatory noncompliance or a costly breach.

[ Learn more: Software Supply Chain Risk Report: Tools Gap Leaves Orgs Exposed ]

How supply chain security debt builds up

Tim Molden, CIO of global executive engagement for Tanium, said technical debt is the result of trade-offs that have been made over time, either intentionally or without choice. 

"Weaknesses in the software supply chain are often attributable to such decisions. The way most companies manage technology generally favors the gradual increase in technical debt, and consequently more exposure in the software supply chain."
Tim Molden

This is not a blame-the-developer story. Most devs contribute to technical debt due to business decisions made for them. They're just trying to make the best decisions they can under the time and budgetary constraints put in front of them.

Molden shared the example of the decision in a budget cycle to prioritize a business innovation technology over an investment in the underlying infrastructure. Such a business decsion can result in an exposure whereby the modern application code does not adequately compensate for something in the legacy infrastructure it is running on.

"Software developers are always having to compensate for sub-optimal conditions in which the software is going to run and it’s easy to overlook something."
—Tim Molden

The old rule of "fast, cheap, good — pick two" applies as strongly as ever to software developmemt today, with security functionality falling under the "good" header. When the business tells them to, though, development teams will consider "good enough" to be known insecure functionality — if they can still tick off the fast and cheap checkboxes, Barratt said.

"So you’re shipping product fast, getting releases out ahead of time but then suddenly are a victim of your own excessive success in getting product to market and find that you’re not spending the necessary time refactoring code, threat modeling the threats across libraries and your minimum viable product is now production and has more Band-Aids than a box of Band-Aids." 
—Andrew Barratt

Matt Rose, field CISO for ReversingLabs, said organizations that have built out aggressive DevOps and CI/CD programs without a strong application security component are struggling more than ever to keep up with the volume of security issues that keep rearing their heads. Rose said the "victim of their own success" situation was common on agile teams that depend on and reuse of open source components as a way to further accelerate their output.

"Applications are getting released faster than ever and they're more complex than ever, and that speed has given limited time to address the technical debt that you're actually creating."
Matt Rose 

The burden of supply chain security debt is heavy. The recent Software Supply Chain Security Risk Report, based on a survey conducted by Dimensional Research of 321 IT professionals and published by ReversingLabs, found that nearly nine in 10 practitioners have detected security issues in their supply chain in the last year. And 88% say software supply chain security presents enterprise risk to their organizations.

Rose says software development is like a social media-driven platform, where everyone wants the latest and greatest features, the best look-and-feel, and the newest capabilities to keep their users and business stakeholders happy and interested in the product. But all of that some at the sacrifice of technical debt—security or otherwise.

"They're like, well, we'll get to that at some point. Usually the feature-function debt gets paid down more than the security stuff, because functional issues are more apparent and likely to be iterated on. And so teams keep pushing down the technical debt associated with security patches, vulnerabilities, or supply chain elements."
—Matt Rose 

Pay down security debt principal vs. interest

Most organizations today are unfortunately only barely paying down the interest of software supply chain security debt. They engage in reactive vulnerability management or isolated app sec testing, but they never quite start paying down the principal by addressing underlying issues that create debt. One example Rose gave is that they'll seek out an individual critical CVE over and over again across the application portfolio, without ever recognizing that it keeps appearing because of heavy dependency on a single component that's reused again and again across the supply chain.

"It's a balloon mortgage and you're only paying the interest — you're not paying the principal at all."
—Matt Rose

Organizations have to get smarter about tackling the problems that incur the debt in the first case. Of course, this starts with underlying fundamentals like developing with Security by Design principals, such as those advocated by CISA.

David Lindner, CISO for Contrast Security, said that putting product security teams in place are a core strategic building block for establishing that long-term Security by Design ethos.

"Security by Design emphasizes integrating security considerations into every phase of a product's lifecycle, from its initial design and development to deployment, maintenance, and eventual retirement."
David Lindner

In the immediate, in-the-weeds tactical terms, there is also the matter of paying down existing security debt, said Barratt. The most dramatic tactical moves to pay down security debt are very specific to the product at hand, but there are some key places to look first. 

"This is highly product-specific. I would spend time looking at interface boundaries, typically where data is or can be consumed between libraries or even between systems where there is a risk that manipulation could take place and unintended results happen leading to security vulnerabilities."
—Andrew Barratt

Establishing a mature supply chain security program that pays down security debt on a regular basis also requires taking holistic and systematic approach to evaluating threats across the entire supply chain, he says.

"Systematic understanding of everything and the ability to systematically evaluate threats across your software supply chain are important. Software Bills of Materials (SBOM) are helpful here, but really tools are needed that identify all the threads of an applications supply chain so that fixes can be applied and mass analysis can be performed quickly."
—Andrew Barratt

ReversingLabs' Rose said that organizations too often are only looking at a few pieces of the 10,000-piece jigsaw puzzle that is today's software supply chain risk.

"I don't think as an industry we're really looking at that whole picture. And that's where the technical debt really comes in, is you're just looking at pieces. As an industry, we're still looking at pieces of the puzzle, the alphabet soup of application security testing solutions of SaaS, DaaS IaaS, SCA, RASP, API scanning are all looking at a certain area or fraction of the application."
—Matt Rose

The Software Supply Chain Risk Report noted that 74% of IT and security professionals say those fragmented app sec tools — SAST, DAST, and SCA — aren't adequately protecting organizations from supply chain threats.

Rose explained that organizations need to work smarter to pay down debt by focusing on the security issues that are both likely to be exploited and pose the highest impact based on how they're implemented within an application.

"The supply chain risks that you're talking about don't really propagate themselves until the whole package is put together. And that's what I'm really screaming about, is how are you testing? You have to be able to take an educated guess based on time and resources available to measure the impact and likelihood to focus the attention on the technical debt that is most important to security risk."
—Matt Rose

Article Link: Supply chain security: Is technical debt weighing your team down?