Advanced cyber threats have become more sophisticated, leading to an evolution in the tools used to combat them. SIEM, which has been around for over 20 years, has significantly advanced its capabilities over time. Initially reliant on basic pattern-matching and threshold-based rules, modern SIEMs have improved their analytic abilities to tackle more sophisticated threats. This evolution is known as the ‘Detection Maturity Curve,’ demonstrating the shift from simple alert systems to advanced mechanisms capable of predictive threat analysis. However, modern SIEMs still face challenges scaling for large data sets and long-term trending or machine learning detection, which impacts an organization’s ability to respond to complex threat actors.
To address these challenges, Databricks provides a unified analytics platform powered by Apache Spark™, MLflow, and Delta tables, enabling scalable and cost-effective big data and machine learning solutions for cybersecurity teams.
The blog post outlines a journey of building evolving security detection rules, transitioning from basic pattern matching to advanced techniques. It details each step and highlights how the Databricks Data Intelligence Platform has been utilized to run detections on over 100 terabytes of monthly event logs and 4 petabytes of historical data, setting a global record for both speed and cost efficiency.
The main goal of the post is to demystify the detection patterns described in the detection maturity curve and explore their value, benefits, and limitations. To facilitate this, a GitHub repository has been created, containing the source material for the blog and a helper library with repeatable PySpark code that can be used for cyber analytics programs.
The blog post then delves into different types of SIEM detection rules, providing examples and code for each:
1. Pattern-Based Rules: These are the simplest form of SIEM detection, triggering alerts upon recognizing specific patterns or signatures in data. This can be used for organizations in the early stages of their cybersecurity program or those facing well-documented threats.
2. Threshold-Based Rules: These rules trigger alerts when events surpass predefined limits or thresholds, useful for scenarios like monitoring network traffic or tracking login attempts.
3. Statistical Anomaly Detection: These rules spot deviations from ‘normal’ behavior and are ideal for mature cybersecurity environments with extensive historical data.
4. Trending-Based Rules: These rules identify anomalies or significant changes in an entity’s behavior over time, adept at uncovering subtle, evolving threats. They require ongoing analysis of large volumes of data.
5. Machine Learning-Based Rules: The most advanced detection rules leverage machine learning algorithms to adapt to threats, often catching attacks that more deterministic rules might miss. Operationalizing machine learning models requires significant expertise and platforms, which Databricks facilitates.
By outlining and explaining these detection rules, the blog post aims to provide practical insights and guidance for cybersecurity teams in implementing effective threat detection strategies.