Metrics are great, but I prefer the ones that inspire action.
I’ve been thinking about this lately through the lens of detection engineering. There are various metrics that help quantify a detection’s performance, from volume and false positive rate to precision and recall. They can be helpful for identifying a detection that isn’t catching evil often enough, but identification is just the first step. The subsequent action — how to improve it — is what I’m most interested in.
So let’s say we’ve found a detection that stinks. The SOC has complained it’s noisy and the metrics agree. What do we do?
Well, it depends on how deep you want to go. There are layers.
At the top is a binary decision that I despise:
A. Do nothing.
B. Disable the detection.
Unless you’ve already done due diligence with data analysis, this should never be the start and end of your decision tree, so let’s ignore that lazy option and go a layer deeper.
If we’re going to do something and not just disable it, that leaves us with improving it. Where do we begin? For now we’re going to focus on reductive tuning (I just made that term up). To me, that means we are looking to add exclusions (or filters) to our existing detection logic to refine the results returned. Another approach could be augmentative tuning (yes, I just made that up, too), which would involve enrichment and/or additional logic for more finely capturing the intended activity.
We could talk to our friends in the SOC to hear their anecdotal observations of the detection, and that could help point us to a valid starting point. But, I prefer to look at the data.
The data we should gather are the historical alerts the detection has emitted, including all normalized evidence.
From this dataset, we will seek to identify the evidence contributing most to the volume — or what I like to call, prevalent evidence. Depending on your tooling, there may be a user interface that can assist us. For example, in Splunk, clicking on the Fields sidebar can show us the top values and their corresponding %.
https://docs.splunk.com/Documentation/Splunk/9.4.1/SearchTutorial/UsefieldstosearchIf you’re using Pandas, you can do something like this
df[‘src_user’].value_counts(normalize=True)
Using this technique, we may find some very prevalent evidence in these results. We may be tempted to immediately tune this value out, but that might be a little overzealous. We don’t yet know if this value is always benign, or if it’s possible to be present during malicious activity.
How can we mitigate this risk? Let’s go another layer deeper.
Instead of picking off prevalent evidence independently, what if we could find combinations of evidence that are prevalent. Instead of just wacking all activity that is performed by a specific user (risky!), what if we found that a large percentage of the alerts are from a specific user on a certain host? That’s a bit better, but still not great.
But, we have more layers!
What if we could determine that most of the activity is performed by the same user, on the same host, executed by the same parent process, with the same command line arguments? Now that is a tactical tune!
With high prevalence of that large a combination (4 items), the likelihood of that grouping to be observed in a true positive is reduced, but we should still double check this isn’t actually a thing. Once we’ve confirmed, excluding that group of key/value pairs is bound to improve our detection’s efficacy. Read that again. We’re excluding the key/value pairs, not just OR’ing some strings that may be present in other fields which would inadvertently exclude activity we still want to see!
I’ve long desired a method to compute and visualize these evidence groups efficiently. I’ve got something in the works that I hope to share soon.
In the meantime, I’m curious — how do you tune detections and what tooling assists you?
Tactical Tuning by Combinational Reduction was originally published in Detect FYI on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.
Article Link: Tactical Tuning by Combinational Reduction | by br4dy5 | Apr, 2025 | Detect FYI