An Empirical Study on Using Large Language Models to Analyze Software Supply Chain Security Failures

by   Tanmay Singla, et al.

As we increasingly depend on software systems, the consequences of breaches in the software supply chain become more severe. High-profile cyber attacks like those on SolarWinds and ShadowHammer have resulted in significant financial and data losses, underlining the need for stronger cybersecurity. One way to prevent future breaches is by studying past failures. However, traditional methods of analyzing these failures require manually reading and summarizing reports about them. Automated support could reduce costs and allow analysis of more failures. Natural Language Processing (NLP) techniques such as Large Language Models (LLMs) could be leveraged to assist the analysis of failures. In this study, we assessed the ability of Large Language Models (LLMs) to analyze historical software supply chain breaches. We used LLMs to replicate the manual analysis of 69 software supply chain security failures performed by members of the Cloud Native Computing Foundation (CNCF). We developed prompts for LLMs to categorize these by four dimensions: type of compromise, intent, nature, and impact. GPT 3.5s categorizations had an average accuracy of 68 report that LLMs effectively characterize software supply chain failures when the source articles are detailed enough for consensus among manual analysts, but cannot yet replace human analysts. Future work can improve LLM performance in this context, and study a broader range of articles and failures.


page 1

page 2

page 3

page 4


A Categorical Archive of ChatGPT Failures

Large language models have been demonstrated to be valuable in different...

Simulating the 1976 Teton Dam Failure using Geoclaw and HEC-RAS and comparing with Historical Observations

Dam failures occur worldwide, often from factors including aging structu...

Software supply chain: review of attacks, risk assessment strategies and security controls

The software product is a source of cyber-attacks that target organizati...

S3C2 Summit 2023-06: Government Secure Supply Chain Summit

Recent years have shown increased cyber attacks targeting less secure el...

Large Language Models for Supply Chain Optimization

Supply chain operations traditionally involve a variety of complex decis...

S3C2 Summit 2202-09: Industry Secure Suppy Chain Summit

Recent years have shown increased cyber attacks targeting less secure el...

Three Ways of Using Large Language Models to Evaluate Chat

This paper describes the systems submitted by team6 for ChatEval, the DS...

Please sign up or login with your details

Forgot password? Click here to reset