Building compound, high-fidelity detection available for all
Data prevalence in a SIEM context means how common or rare a specific artifact or behavior is within your environment’s data.
Stick around to understand why this is a game changer, not from a marketer or industry analyst, but from a real practitioner.
Baselines & Data Prevalence
Wait, what's the relationship here? Those term are very well connected because baselines are built from prevalence measurements over time.
In practice, prevalence measures frequency of occurrence of something across logs and systems.
- Prevalence tells you how often something happens.
- A Baseline uses those frequencies to define what is normal.
The frequency with which any of the following entities appear within an environment (e.g., in SIEM logs) can be used as a prevalence measure:
- External domains contacted by hosts
- Child processes executed by a LOLBin
- 3. File hashes or binaries
By the way, did I mention I love building baselines? You can too!
Why is that a game changer?
Data prevalence is perhaps the simplest yet most elegant way to incorporate statistical reasoning into threat detection and incident response.
In traditional SIEM detection, rules often look like:
"Alert if [suspicious behavior] occurs."
But modern environments generate massive noise. Attackers blend in. Static rules create alert fatigue. And the list goes on and on — I'm not here for another long SIEM rant you read every week on LinkedIn.
Prevalence changes the game because it asks:
How common is this behavior in my environment?
Instead of simply relying on:
Is this behavior suspicious in theory?
A real attack scenario is far less common than legitimate activity, and defenders can take advantage of that imbalance.
Despite fileless malware and LOLBins, attacks will eventually leave artifacts on disk, creating an opportunity to leverage data prevalence for detection.
How that looks like in practice?
I have been helping startups design alert frameworks and refine product roadmap for some time now.
Needless to say, incorporating data prevalence is one of the killing features for any modern SIEM or Security Analytics product. It's a MUST have.
But without clear use cases, it becomes largely wasted potential.
Here’s the point: pairing strong UI/UX with such a powerful feature can create a significant competitive advantage.
Today, there's only one top-tier product doing that: MS Defender XDR.
I’m not being paid to write this — I genuinely hope to encourage product teams reading this to incorporate such valuable feature into their own platforms.
I write KQL almost weekly and it’s a great experience. However, Defender still has significant room for improvement in the UI/UX domain.
While it’s not applicable to every entity, Microsoft provides a function called FileProfile() that enriches logs with a Global Prevalence value, reflecting how common a file appears to be across its customer base.
That's a major data point for defenders, especially when it comes to an artifact involved in nearly all endpoint detections.
MS Defender: Global Prevalence available for all!
In fact, any MS customer with access to Defender's KQL query editor can check the number of instances of a file observed by Microsoft globally!
For instance, let's grab a fresh malware binary from MalwareBazaar:
SHA265 hash value from a recent malware sample (AsyncRAT)Here's what one can see after invoking the file profile in Defender:
GlobalPrevalence value indicates this file was seen only once across all defender tenantsBuilding a Resilient, Compound Detection
If you follow my articles long enough, you've probably came across the concept of Detection Opportunity Cost.
Among the various insights I share, the main takeaway is that strong, reliable custom detections always start with a well-defined use case.
And what better way to validate that than through a real incident or a purple team exercise?
After working on numerous enterprise projects where I help customers design and built detections following real incidents, one recurring pattern consistently stands out: artifact prevalence.
I mean, low prevalence.
That's when combining pattern and behavior matches with low artifact prevalence makes into a great compound indicator.
Use Case: DLL loaded from a low=prevalent userland app binary
The following template can be used in MANY, really many detections while generating very few hits given the constraint on prevalence:
DeviceImageLoadEvents
// Load relevant events happening within the last 7d
| where Timestamp > ago(7d) and ActionType == @"ImageLoaded"
// Focus on binaries with no file metadata
| where (isempty(InitiatingProcessVersionInfoCompanyName) and isempty(InitiatingProcessVersionInfoProductName))
// Also located in users local apps folder only
| where InitiatingProcessFolderPath matches regex @'(?i)\\users\\.+\\appdata\\local\\apps.+\.exe$'
// Filter in userland DLL file loads only
| where FolderPath matches regex @'(?i)\\users\\.+\\appdata\\local\\apps.+\.dll$'
// Aggregate and calculate extra stats
| summarize DeviceCount=dcount(DeviceId), DLLCount=dcount(FolderPath), DLL=make_set(FolderPath), arg_max(Timestamp, *)
by InitiatingProcessSHA256, InitiatingProcessFolderPath
// Narrow to the ones loading a single file and seen from one device only
| where DeviceCount == 1 and DLLCount == 1
// Load the prevalence profile
| invoke FileProfile(InitiatingProcessSHA256, 100)
// Filter in only very low prevalent ones
| where isempty(GlobalPrevalence) or GlobalPrevalence <=2
The last line filters out most of the noise given that it will only output very low-prevalent binaries.
The Device and DLL counts work as an ephemeral baseline (of last 7 days) and increase the chances of catching suspicious stuff.
Use that as a prototype to build baselines persisted on disk addressing longer periods and you make it even more effective.
There might be some extra fine-tuning needed there (secret sauce) but I'm sure you will get inspired with that already!
What else could Threat Detection product and engineering teams do?
What Defender currently offers is only a small glimpse of what’s possible. Platforms like Splunk, for example, provide multiple ways to define and leverage custom baselines.
If you’re building a new detection product, consider UI/UX as a core capability that enables users to easily leverage these features, rather than treating it as an afterthought.
Follow me for more insights, and feel free to reach out if you’d like help advising or accelerating your product or practice teams.
Until next time.
Written by Alex Teixeira, Detection Engineering & Security Analytics SME
Don’t forget to subscribe and follow me to get notified about new stories!
Data Prevalence: The Game Changer in Threat Detection & IR was originally published in Detect FYI on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.
Article Link: https://detect.fyi/data-prevalence-the-game-changer-in-threat-detection-ir-f1162de0b6b3?source=rss----d5fd8f494f6a---4