Everyone seems to be obsessed with False Positives right now in Security. But this is a solved problem in other industries..
How does a fortune 500 company evaluate thousands of feature changes in their product, across thousands of customers, in real time, continuously… over data that keeps changing..? Some of them have data scientists that use Statsig, but all of them use some kind of a statistics engine that evaluates performance against costs and risk.
How does a Hedge Fund choose to take the best performing prototype investment algorithms? They give them real data first, and then they use statistical analysis and optimize the results relative to the costs and risks involved, in order to screen for chosen models.
What about running thousands of hypothesis tests across incoming data for disease prediction, then deciding which hypothesis tests discoveries to send to a team of experts to investigate?
All of these have a few characteristics similar to the modern MSSP SOC.
- They run multiple hypothesis tests, continuously, across newly seen data. So each execution brings a unique test.
- The occurrence of True Positives is rare, so most findings are False Positives.
- They have fixed resource constraints limiting the number of investigations that the experts can complete, vs number of tests that can be patched/updated/engineered. So there are cost limits.
- The impact from missing True Positives is commercially high, so care is taken to quantify and measure risk.
To any data scientist/statistician/economist/operations researcher..
a) how many expert-hours do we have? (SOCs can answer this)
b) how big a queue can we clear? (SOCs can answer this)
c) how much noise can we actually stomach? (SOC’s don’t know how to articulate this).
d) how much risk of missed True Positives do we face? (This is why SOCs haven’t solved for cost minimization yet).
So if this optimization problem has been solved in other areas (some with negative impacts being loss of life..) then why is it that SOCs keep saying they are drowning in False Postives?
“The SOC is a screening operation running against a fixed budget of expert hours, but it treats itself as a unique problem child, because it has unmeasured uncertainty on how to manage it’s risk of missed True Positives.” — me calling it out.
If your MSSP is chasing “FP reduction”, then it’s already failed to frame the problem correctly, and it will always have an optimization problem unsolved.
You want to keep the noise inside a budget, for the least cost to business, without throwing away True Positives. But in order to do that we need to measure risk of lost True Positives..
So how do we do that?
Lessons from the GUIDE Dataset
A few years ago, Microsoft ran a competition on Kaggle, to help find data science solutions “that help security operations centers identify, prioritize, investigate, and respond to cybersecurity incidents”.
They took a very large sample of incident data from their customer base, and provided it as the GUIDE dataset.
GUIDE is the largest ground truth labelled incident dataset out there to date. It is composed of 707,000 graded incidents across 7,219 detectors and 5,340 tenants, every one labelled true-positive/benign-positive/false-positive, timestamped, and tagged with a MITRE ATT&CK tactic.
For my analysis, I want to investigate what a MSSP with multiple customers looks like in the GUIDE dataset. So I focused on the 857 that generate incidents everyday over a 12 day period. This is because in my experience, if a SOC has a lot of FPs then it’s usual to see most of the daily incidents come from the same noisy customers or detectors. The dataset also validates this, but we’ll get there soon.
Lesson 1: Class Definitions MATTER.
GUIDE provides labels for incidents as True Positive, Benign Positive and False Positive. Those are the Microsoft labels, but many other tools use them too..
But someone with a bit of data science experience will challenge this..
Statistician — “hmmm? What is Benign Positive? Is that meant to be a class? Is this a multiclass confusion matrix?”
“Nono, it’s a security industry added gem! It means the test ran well, but the outcome wasn’t a True Positive! :)” — Security Industry
Statistician — “em.. so it’s not a TP, so it’s a FP?”
“Huh? its not a TP and not a FP dude” — Security Industry
Statistician — “Where does BP sit in all of the time-tested statistical measures for classification performance?”
“I don’t know. It’s built into my SIEM platforms.. so?” — Security Industry
Statistician — “I mean, there’s some argument towards BP being a separate class, but it’s generally not used when we’re dealing with rare TP event discovery”.
“So what do we do then? How would this impact our approach? — Security Industry”
Let me show you.
I believe that the industry has set itself up with a fundamental problem. Why does the SOC exist (outside of making profit)? It’s there to provide assurance to the customer that their environment is being protected from unwanted actions, detecting and remedying any that occur. That means that the class definitions should to be aligned to yield. When this is done, the definition of the ‘Benign Positive’ splits into two, one that belongs in true positive, and one that doesn’t. If you want to understand this more from a stats angle, you can read Fawcett, “An Introduction to ROC Analysis” or ask your favorite LLM..
So the real definitions at play are more like this (imo):
True Positives
a. [TP] Malicious / Unwanted Activity. Genuine threat or policy-violating activity, SOC cost to investigate is justified. This is why the SOC exists.
b. [TP] Client Useful / Assurance. Not an attack, but the finding delivered value. E.g. misconfiguration, shadow IT, control failure. The cost justified. This is also why the SOC exists.
False Positives
a. [FP] Benign. No threat, no value delivered. Time spent, nothing returned to the customer. SOC incurs a cost, negative value to customer. It doesn’t matter if it’s a ‘suspicious process’ or a ‘anomalous X doing Y’, if it’s triaged and not within the two TP classes, then it’s a FP.
b. [FP] Noise. No threat, no value delivered again.. Time spent triaging, and not even escalated to the customer or next analyst in escalation path.
False Negatives
d. [FN] Missed Malicious/Unwanted Activity. These are the genuine threat or policy-violating activties that your detection rules did not capture, that you later found out about. Penetration test reports, DFIR reports you review against, dare I say.. Real incidents?
e. [FN] Missed incidents via rule review. These are the FNs found from reviewing your rulesets against the Detection Weakness Enumeration standard.
True Negatives
Most don’t bother defining this, but I tend to treat the size of the search space as as True Negatives, mostly because I’m sneaky and it makes you look less bad when you need the counts for your performance metrics. But you shouldn’t need to worry about it for the cost optimization problem in your SOC, because it’s somewhat useless for this. And if a customer needs a metric that needs the TN count, then you should provide it and provide them your confusion matrix too.
Two things you might have noticed:
- The two TP determinations focus on maximizing value to the customer, aligned to the purpose of the SOC. This includes detections from penetration tests, in [TP] Client Useful / Assurance.. It’s the portion of the “Benign Positive” earlier that points to yield.
- The [FP] Benign class results in a cost to the customer, so it’s separate to the noise. This helps the SOC track it’s reduction.
- I included False Negatives. That’s because it’s a very real class that does actually exist. Are you tracking these???
Ok. Now that we have a definition of TP and FP, etc we will make all the Benign Positive labelled alerts in the GUIDE dataset False Positives, because that’s the risk adverse thing to do.
Lesson 2: ‘Shit-In-Shit-Out’ is an unbiased universal law.
When we investigate where the false positives come from, we see that appx 1% of detectors contribute around 80% of all FPs…
The grey dotted line is the line of perfect equality, meaning that if all the detectors equally contributed to the volume of False Positives, then we’d see that line. The yellow vertical line shows a share of 10% (more later on this). When you see red and blue lines having a large prevalence of FPs, then we know that there are a few poorly designed detectors that are contributing the bulk of your FP pain.
But what if they are caused by a few very noisy customers? Remember when I said I reduced the dataset down to those organizations that were active every single day of the 12 days? (reduced for my proposed stats engine, ‘always-on core as the mock MSSP customer base’).
The above figure shows us that most of the incidents in the dataset were generated by organizations that produce incidents every day. But when we remove those organizations that only generated incidents once, we see that the skewedness of the noise to detectors doesn’t really change (below figure).
But we know this already, why don’t we just cut them out?
At 80% total FPs, the ~1% of detectors are the noisiest but are also the highest-volume, highest-recall rules. So a blunt “delete the noisy rules” cut removes ~all the noise loses 90% of true positives..
Lesson 3: True Positive Risk can be taken from GUIDE! ❤
When we look at the distribution of the gaps in time between each True Positive, we see that it’s very very dense at 1.3 minutes.
This makes sense, attackers think in graphs, if we turned off a rule we’d probably still see some around it. The dataset shows us that TPs arrive in tight clusters and 98% of gaps are under 4 hours.
But knowing that, what if once a detector is flagged for a patch, and it gets taken down, what RISK do we take on? What is the probability of MISSING a true positive incident while the detector is down? How does that change over time?
If we park a rule forever (i.e cut it) then we lose 40% of all FUTURE TPs. But if we patch it within the first 4 hours, we’d only lose 1.3% of true positives. Within 1 hour is 0.3%, which is pretty low risk. As we delay the patch, the increasingly more likely it is that we’ll miss a True Positive incident. Hey, this means we have numbers to optimize against guys.
But remember when we saw that detectors to FP volumes had a very strong skew? It turns out that at a 4 hour SLA to patch a flagged detector, 240 detectors of 331 flagged actually miss true positive incidents.
So how do we use these lessons to solve the cost optimization problem? For this I propose Dynamic Incident Routing Engine or DIRE for short.
- Penalize the detectors that perform badly, do this continuously via betting e-values
- Route the flagged detectors based on their type of badness
- Do this informed by cost limitations for both patching and triage, and risk for missed TPs.
Applying Dynamic Incident Routing Engine (DIRE) the Stats Engine I propose, to GUIDE.
DIRE is a statistical process-control layer for the SOC. Just like the other industries have. It sits between the detection platforms and the SOC and only forwards incidents that statistically carry informative value.
How it works
There are two parts. First, the SOC Manager decides what thier tolerance for False Discovery Rate is. That’s FP/FP+TP. It tells you how much of the incidents can be tolerated as False Positive. For DIRE we model using 10%. 10% is pretty generous.
The second part is the stats engine itself.
Every hour, DIRE re-asks three questions of each rule:
- Q1: Are you globally bad? This detector’s noise rate vs all other detectors on the platform. If it’s shit across all customer environments, then it’s globally shit.
- Q2: bad for this customer? One tenant’s slice of the detector vs the same detector’s other tenants. If it’s only shit for a handful of customers, and not globally, then maybe the customer’s environment changed?
- Q3: spiking now? The last hour vs this rule’s own trailing baseline. Q1 and Q2 are lagging measures, if a detection rule suddenly flags the SOC, then it needs to be patched.
DIRE measures evidence with an e-value. A detection rule is tested continuously, forever, on data that keeps growing. This means there is a “peeking” problem that invalidates p-values. E-values are built for exactly this reason.
An e-value is actually a betting score. It’s how many times better the hypothesis “this rule is worse than its reference” predicted the data than “this rule is normal.” An e-value of 10 means 10-to-1 evidence. DIRE flags at e ≥ 1/α, and with α = 10% (the FDR tolerance rate mentioned above) that’s a 10:1 limit. Because “a fair bet against a true null can’t grow your money on average (E[e]=1)” a simple threshold gives a guaranteed ≤10% false discovery rate. That means you can scale up and down in your SOC modelling, and force your tolerance to be a constant. That has large implications in the commercial modelling benefits I’ll show later.
DIRE creates a matrix (rows vs columns) of detectors vs customers, in order to asses whether customer environments have impacted the detector performance.
- A tenant with 5 false positives in 6 alerts isn’t “83% noise”, it’s unknown. Due to this DIRE shrinks every thin estimate toward the level above it (tenant-cell → detector → family → whole estate) using empirical-Bayes, after subtracting the sampling-noise component. Without this, tiny samples throw extreme rates that skew any ranking.
- Each rule is judged against a reference that excludes itself (leave-one-out sampling). Otherwise a bad rule ruins the baseline it’s being measured against.
- Q1 runs first, a globally-flagged detector skips its per-tenant tests as one MSSP-wide bad rule shouldn’t spawn fifty per-tenant tickets, and GUIDE data taught us that globally badness creates most of the FPs. And once anything is flagged it leaves both the queue and every future baseline, so one noisy rule can’t drag its neighbours’ references down with it.
- Every decision is auditable. Each step writes a ledger row per tenant-cell, counts, shrunk rate, the reference it was judged against, all three e-values, and the verdict. You can always answer “why was this flagged?”, or “why wasn’t it?”.
DIRES output is a routable work-ticket with a costing. A flag emits a tuning task carrying the evidence, the reference rate, and a cross-tenant distribution, plus a 24-hour look-ahead risk score of lost TPs. It also outputs cost information: recurring analyst-hours, one-time engineer-hours, payback, and true-positives-at-risk.
OK. So we’ve established that DIRE is a statistics engine that continuously bets on each detector based on it’s TP/FP performance, routs a work ticket, enables cost optimization, gives a business risk for the detector etc.
For this example, I will apply DIRE to the 12 days in the GUIDE dataset. The following assumptions apply.
- The time to patch a globally bad detector is 16 hours. I think this is realistic, as it’s calendar time (not engineer time). A from scratch detector needs backtesting and emulation, so 16 hours is ok imo.
- The time to patch a locally bad detector is 4 hours. This means that the FPs coming in for the hour step for a detector:customer/organiztion flag as low informative value to the customer, so the SOC reviews FP notes and sees that there is an opportunity for a local exclusion/automation rule for the customer. 4 hours is enough time, the lag affect of DIRE means the FPs have SOC analyst notes included.
- The time windows above are based on the TP miss probabilities from GUIDE.
- DIRE needed 7 days of data to inform baselines, so the remaining 5 were the test window.
First, RISK. How does DIRE perform over the 12 days with risk?
When we consider that attackers think in graphs, and we would see other alerts inform a campaign, we can construct a risk score around campaigns.
Yes, I used claude to help generate these figures. So what?When we considered a 16h and 4h patch windows, we lose 2.3% of campaigns. That was due to the large patch window sizes for globally bad rules. That reduces as we reduce the patch window. It may not be ideal, but it’s data we can incorporate into our planning in the SOC around how fast to patch, argue for resources for faster patching etc etc.
The figure below shows us that the lost TPs are from the globally bad detectors, not quite from the local cell-level patches. That makes sense due to the skewedness of the FPs to the detectors.
But this is risk from the GUIDE dataset, it doesn’t mean that a True Positive will occur. The figure below shows us where these risks occur in the MITRE ATT&CK tactic level. The bulk of this risk is initial access based, so if customers enforce strong Conditional Access policies and App Policies (if entra) and phishing resistant 2FA, the risk has a mitigation to help support the larger patch window.
In fact, from the DIRE simulation we can see that 71% of the risk is introduced by the top 10 volumous detectors. That further supports the need to emulate your highest performing detectors if they are also very noisy. You’ll notice that, these risks are much lower than if we were to straight cut the detectors out. That’s because we DIRE’s patch windows allow for time to better emulate detectors and catch TPs.
So now we know the risks, the next question to ask is the benefit that taking the risk leads to, and what the costs are that offset the benefit.
Finding 2, Continuous FP reduction benefit.
In DIRE we assume that an incident takes 20 minutes to triage. We assume that every patch requires an engineer time. Although we don’t provide the analysis in a dollar value, but a time unit, it wouldn’t be difficult for a SOC manager to take DIRE outputs and apply them to a cost model.
The figure above shows us that once DIRE starts flagging incidents for patching, running tests each hour, that the prevalence engine (Q1–3) drop false positives from 82% to 59%, with the FDR tolerance limit further dropping it to 22%, closer to the 10% budget. We don’t see the 10% being meet post 15th June, but if we had more data we would have seen it trend downward as it’s a lag model.
The gradual reduction post the immediate drop shows that the top worse performing detectors that skew the FP volumes get flagged almost immediately, with consecutive reductions occurring due to the prevalence tracker (the e-values).
Fixing the top 10 flagged detectors recovers in benefits 60% all avaliable recoverable cost. That means that a short burst of engineering effort solves most of the pain issues. We kind of knew that already though right? Glad the data shows us this.
These hundreds of hours represents the volume of detectors that GUIDE dataset has, so in practice it’s likely even smaller.
When we consider the engineering cost as a one off for a detector, and the savings to be reocurrong we see that one we factor in the risks from GUIDE data, the benefit is huge. We just needed to know the risk. The figure below shows us that as long as the TP risk has mitigations (discussed above) prior to patch, then there is little argument to keep a detector running prior to a patch starting.
In fact, when comparing an identify and cut approach to DIRE, we find that DIRE removes 96% of all False Positives (of cut rules), and only lost 5% of the True Positives in the GUIDE data (because they happened to occur within the patch windows).
And if we really can’t tolerate that much risk, even post mitigation steps, then we can directly reduce it by shortening the patch window for globally bad detectors, which are the contributors to the risk, as demonstrated below on the left (flagged cells are the detector vs organization pairs).
The return efficiency is pretty good. If we can accept the risk, then we should implement DIRE.
Finding 3, Patch TODO ticket routing and SOC scalability
DIRE’s two step method of FDR tolerance and evalue show that the long term benefits/savings of DIRE can be found in the local patches within customer environments with 25% local savings from the FDR tolerance of 10% and 16% from the evalue tracking.
When a SOC has an environment that is tuned, and platform teams can attribute SOC savings to revenue, and there are SLAs on the types of patching that DIRE raises, then we see an almost uniform saving of 80% all FPs over each SOC shift, where actually the evening shift sees more of the residual queue than the day shift. This isn’t surprising, as we saw earlier that most of the FPs peaked during business hours.
Once this level of control, performance evaluation can be achieved in a SOC, then it is easier to scale up and down and know the cost impacts.
Final Remarks
The SOC isn’t a unique problem child. It’s a screening operation, thousands of continuous tests, rare true positives, a fixed budget of expert hours, a high cost for anything missed.
Other industries solved this by measuring the cost of the noise against the risk of the miss. Security chased “FP reduction” instead, which is a target with no risk articulated, so set your eyes on optimising to risk.
GUIDE data gives us a prior for risk. TPs arrive in tight clusters, so the cost of parking a detector is function you can read off the data. 40% of future TPs if you cut it forever, 1.3% if you patch inside 4 hours, 0.3% inside one hour.
DIRE is just the stats engine that spends the cost deliberately to get 96% of the FPs of a naive cut, at 5% of the TP loss. That’s an informed case for any for-profit-SOC.
Source code coming soon.
Online False Positive Reduction via Statistical Process Controls was originally published in Detect FYI on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.