Time for aCTIon: Automated Analysis of Cyber Threat Intelligence in the Wild

by   Giuseppe Siracusano, et al.

Cyber Threat Intelligence (CTI) plays a crucial role in assessing risks and enhancing security for organizations. However, the process of extracting relevant information from unstructured text sources can be expensive and time-consuming. Our empirical experience shows that existing tools for automated structured CTI extraction have performance limitations. Furthermore, the community lacks a common benchmark to quantitatively assess their performance. We fill these gaps providing a new large open benchmark dataset and aCTIon, a structured CTI information extraction tool. The dataset includes 204 real-world publicly available reports and their corresponding structured CTI information in STIX format. Our team curated the dataset involving three independent groups of CTI analysts working over the course of several months. To the best of our knowledge, this dataset is two orders of magnitude larger than previously released open source datasets. We then design aCTIon, leveraging recently introduced large language models (GPT3.5) in the context of two custom information extraction pipelines. We compare our method with 10 solutions presented in previous work, for which we develop our own implementations when open-source implementations were lacking. Our results show that aCTIon outperforms previous work for structured CTI extraction with an improvement of the F1-score from 10


page 1

page 3

page 7

page 8


ThreatKG: A Threat Knowledge Graph for Automated Open-Source Cyber Threat Intelligence Gathering and Management

Despite the increased adoption of open-source cyber threat intelligence ...

Malware Knowledge Graph Generation

Cyber threat and attack intelligence information are available in non-st...

Automated Dataset Generation System for Collaborative Research of Cyber Threat Intelligence Analysis

The objectives of cyber attacks are becoming sophisticated and the attac...

Preserving Knowledge Invariance: Rethinking Robustness Evaluation of Open Information Extraction

The robustness to distribution changes ensures that NLP models can be su...

From Threat Reports to Continuous Threat Intelligence: A Comparison of Attack Technique Extraction Methods from Textual Artifacts

The cyberthreat landscape is continuously evolving. Hence, continuous mo...

On the Integration of Course of Action Playbooks into Shareable Cyber Threat Intelligence

Motivated by the introduction of CACAO, the first open standard that har...

Plumber: A Modular Framework to Create Information Extraction Pipelines

Information Extraction (IE) tasks are commonly studied topics in various...

Please sign up or login with your details

Forgot password? Click here to reset