GraphWeaver: Billion-Scale Cybersecurity Incident Correlation

Overview of the GraphWeaver architecture from the perspective of a single geographic region. Alert telemetry is retrieved from an ADLS system and processed into two PySpark dataframes: one containing the latest 35 minutes of data, and the other the last 2 days. These dataframes are then correlated based on shared entities between pairs of alerts. Subsequently, threat intelligence and security domain knowledge are applied to filter out invalid correlations. The refined correlations are then centralized to the main node and converted into a graph, enabling a minimum spanning tree algorithm to remove redundant incident correlations. Finally, a human-in-the-loop feedback system is employed, where threat researchers review correlation reports to optimize parameters, review assumptions, and pinpoint potential correlation gaps.

Abstract

In the dynamic landscape of large enterprise cybersecurity, accurately and efficiently correlating billions of security alerts into comprehensive incidents is a substantial challenge. Traditional correlation techniques often struggle with maintenance, scaling, and adapting to emerging threats and novel sources of telemetry. We introduce GraphWeaver, an industry-scale framework that shifts the traditional incident correlation process to a data-optimized, geo-distributed graph based approach. GraphWeaver introduces a suite of innovations tailored to handle the complexities of correlating billions of shared evidence alerts across hundreds of thousands of enterprises. Key among these innovations are a geo-distributed database and PySpark analytics engine for large-scale data processing, a minimum spanning tree algorithm to optimize correlation storage, integration of security domain knowledge and threat intelligence, and a human-in-the-loop feedback system to continuously refine key correlation processes and parameters. GraphWeaver is integrated into the Microsoft Defender XDR product and deployed worldwide, handling billions of correlations with a 99% accuracy rate, as confirmed by customer feedback and extensive investigations by security experts. This integration has not only maintained high correlation accuracy but reduces traditional correlation storage requirements by 7.4x. We provide an in-depth overview of the key design and operational features of GraphWeaver, setting a precedent as the first cybersecurity company to openly discuss these critical capabilities at this level of depth.

Citation

GraphWeaver: Billion-Scale Cybersecurity Incident Correlation
Scott Freitas, Amir Gharib
ACM International Conference on Information and Knowledge Management (CIKM). Boise, Idaho, 2024.
Project PDF Blog BibTeX Deployed in Microsoft Defender XDR product Keynote talk at CIKM Industry Day