How to detect Azure Monitor transformation drops before they create blind spots in your security monitoring.
Every hunt, detection rule and investigation starts with one assumption: the data in your workspace shows what actually happened.
But it doesn’t always. It only shows the data that made it through the pipeline.
That pipeline has a filter that many security teams have never reviewed. They may not know how much data it drops or monitor it for changes. If someone changes the filter, important events may stop reaching your tables. No alert fires because the data needed to trigger it is missing. Those events were dropped before they were ever stored.
The good news is that dropped data leaves clues, just not where you would normally look.
This article explains where the filter lives, why it sits outside the SOC’s usual view and which signals you can monitor to catch both malicious changes and simple configuration mistakes.
Understanding Data Transformations
In Azure Monitor and Microsoft Sentinel, data does not go straight from the source into your workspace. It first passes through a Data Collection Rule (DCR).
The DCR decides what data to collect, how to change it and where to send it.
One part of the DCR is the transformation. This is a KQL query that runs on every record as it arrives, before the record is written to a table. Microsoft defines the order clearly the transformation runs after the source sends the data and before it reaches the destination.
Transformations are useful. You can drop noisy data you do not want to pay to store, remove columns you do not need or reshape logs into a common format. Microsoft recommends them for cost control, and in normal use they make sense.
Here is the part of the DCR that matters:
{
"dataFlows": [
{
"streams": [ "Microsoft-Syslog" ],
"destinations": [ "MyWorkspace" ],
"transformKql": "source | where SeverityLevel != 'info'",
"outputStream": "Microsoft-Syslog"
}
]
}The key line is transformKql. It is a KQL query stored in the configuration and every record must pass through it.
In this example, it drops informational syslog messages. That is cheap, sensible and something nobody would question.
But watch what happens when one extra line is added.
The one-line blind spot
Start with a transformation nobody would look at twice. It drops debug logs.
source
| where LogLevel != "DEBUG"
Add one more condition.
source
| where LogLevel != "DEBUG"
| where SourceIP != "10.20.20.20"
Every event from that IP is now gone. It is not hidden in a table you can still query and it is not moved to a cheaper storage tier. It is removed before anything is written.
No detection rule can alert on it, because rules query tables and the record never reached the table.
An attacker using that IP address becomes invisible. And the thing hiding them is just one line in a configuration file.A transformation can quietly remove one column that your detection depends on.
In that case, the events still arrive. The row count looks normal and nothing appears missing at first glance. But the field your rule relies on is blank in every record. The detection still runs, finds nothing and reports that everything is healthy.
The Blind Spot Before the SOC
A normal attacker with admin rights has to take action, generate logs and then clean up afterwards. That cleanup is still an action and it leaves traces. Purging a table, changing retention or deleting records are all things a decent SOC can detect.
A transformation works differently. It acts before the log is created. There is no “after” to clean up.
The evidence is not deleted. It is never stored in the first place. And you cannot write a detection for the deletion of something that never reached your tables.
That is why whoever controls the DCR sits above the SOC not inside it.
The SOC studies the data it receives. The DCR decides which data reaches the SOC. One controls the other and the SOC cannot look upstream by querying its own tables.
You Don’t Need an Attacker to Create a Big Blind Spot
Here is the part that should concern you, even if you never deal with a malicious insider.
A transformation is a KQL query, and queries can have bugs. If the format of incoming logs changes or the query handles a field incorrectly, the transformation can fail. When that happens, the affected records may be dropped.
A careless filter can also remove more data than intended. One wrong operator or one bad assumption about a field can stop an entire category of events from reaching your tables.
There is another trap worth knowing. Transformations only support a subset of KQL. Common functions like coalesce() are not supported. A query that uses them may work perfectly in the Logs blade, but fail or behave unexpectedly once it runs inside the pipeline.
So the query you tested is not always the query you deployed.
Put this together, and you do not need an attacker. You only need one engineer making a small change to a transformation on a Friday to save storage costs and nobody notices until an investigation months later finds nothing where evidence should have been.
That is the real reason to care. A malicious change may be rare, but an accidental mistake is common.
Where Dropped Events Leave a Trace
You cannot see the dropped records, but dropping data is still an event in the pipeline. The pipeline reports on itself, you just need to enable that reporting.
Every DCR automatically creates metrics such as rows received and rows dropped. You can view them in Metrics Explorer without setup. However, to query them with KQL in the AzureMetrics table or see error details, you must enable diagnostic settings on the DCR.
Neither is enabled by default. Until you turn them on, the data needed to catch a silent drop is not collected either.
Open DCR → Diagnostic settings → Enable Metrics + Log Errors → Select Log Analytics workspace → Save → Repeat for every DCR you want to monitor.
Before running the queries below, learn what normal looks like. A busy CEF or syslog pipeline may intentionally drop 30 percent of its rows daily. The rule to watch is the quiet one that normally drops almost nothing.
Run this query once to see where each rule stands.
AzureMetrics
| where TimeGenerated > ago(7d)
| where ResourceId has "DATACOLLECTIONRULES"
| where MetricName in ("RowsDropped_Count", "RowsReceived_Count")
| extend DCR = tostring(split(ResourceId, "/")[-1])
| summarize Value = sum(Total) by MetricName, DCR
| evaluate pivot(MetricName, sum(Value))
| extend DropRatePct = round(100.0 * RowsDropped_Count / (RowsReceived_Count + RowsDropped_Count), 1)
| project DCR, RowsReceived_Count, RowsDropped_Count, DropRatePct
| sort by RowsDropped_Count desc
Signal 1 — A rule’s drop rate jumps above its normal level
This is the main detection. It only fires when a rule starts dropping noticeably more than its own recent history. That helps filter out busy pipelines that normally drop a large amount of data.
let Window = 14d;
let Recent = 1d;
let DCRMetrics = AzureMetrics
| where TimeGenerated between (ago(Window) .. now())
| where ResourceId has "DATACOLLECTIONRULES"
| where MetricName in ("RowsDropped_Count", "RowsReceived_Count")
| extend DCR = tostring(split(ResourceId, "/")[-1])
| summarize V = sum(Total) by MetricName, DCR, Period = iff(TimeGenerated > ago(Recent), "Recent", "Baseline")
| evaluate pivot(MetricName, sum(V), DCR, Period);
let Baseline = DCRMetrics
| where Period == "Baseline"
| extend
Received = coalesce(todouble(RowsReceived_Count), 0.0),
Dropped = coalesce(todouble(RowsDropped_Count), 0.0)
| extend TotalRows = Received + Dropped
| where TotalRows > 0
| extend BaseDropPct = 100.0 * Dropped / TotalRows
| project DCR, BaseDropPct;
let Current = DCRMetrics
| where Period == "Recent"
| extend
Received = coalesce(todouble(RowsReceived_Count), 0.0),
Dropped = coalesce(todouble(RowsDropped_Count), 0.0)
| extend TotalRows = Received + Dropped
| where TotalRows > 0
| extend CurrentDropPct = 100.0 * Dropped / TotalRows
| project DCR,CurrentDropPct, Received, Dropped;
Current
| join kind=inner Baseline on DCR
| extend Jump = round(CurrentDropPct - BaseDropPct, 1)
| where CurrentDropPct > 10 and Jump > 15
| project DCR, Received, Dropped, BaseDropPct = round(BaseDropPct, 1),CurrentDropPct = round(CurrentDropPct, 1),Jump
The two values at the bottom, CurrentDrop > 10 and Jump > 15, are yours to tune after you understand your baseline.
They mean: alert when a rule is currently dropping more than 10 percent of its rows and that drop rate is at least 15 percentage points higher than its normal level.
Signal 2 — A rule starts dropping columns or rows it never dropped before
Row drops show up in the overall volume. Column drops do not, because the rows still arrive as normal.
That makes column stripping the quieter version of this attack. The metric ColumnsDropped_Count/RowsDropped_Count(optional) is what catches it.
//Colums Drop not seen in the past
let Baseline = AzureMetrics
| where TimeGenerated between (ago(14d) .. ago(1d))
| where ResourceId has "DATACOLLECTIONRULES"
| where MetricName == "ColumnsDropped_Count"
| extend DCR = tostring(split(ResourceId, "/")[-1])
| summarize PriorColums = sum(Total) by DCR;
AzureMetrics
| where TimeGenerated between (ago(1d) .. now())
| where ResourceId has "DATACOLLECTIONRULES"
| where MetricName == "ColumnsDropped_Count"
| extend DCR = tostring(split(ResourceId, "/")[-1])
| summarize TodayColumsDrop = sum(Total) by DCR
| join kind=leftouter Baseline on DCR
| extend PriorColums = coalesce(PriorColums, 0.0)
| where TodayColumsDrop > 0 and PriorColums == 0
| project DCR, TodayColumsDrop, PriorColums
//Rows Drop not seen in the past
let Baseline = AzureMetrics
| where TimeGenerated between (ago(14d) .. ago(1d))
| where ResourceId has "DATACOLLECTIONRULES"
| where MetricName == "RowsDropped_Count"
| extend DCR = tostring(split(ResourceId, "/")[-1])
| summarize PriorRows = sum(Total) by DCR;
AzureMetrics
| where TimeGenerated between (ago(1d) .. now())
| where ResourceId has "DATACOLLECTIONRULES"
| where MetricName == "RowsDropped_Count"
| extend DCR = tostring(split(ResourceId, "/")[-1])
| summarize TodayRowsDrop = sum(Total) by DCR
| join kind=leftouter Baseline on DCR
| extend PriorRows = coalesce(PriorRows, 0.0)
| where TodayRowsDrop > 0 and PriorRows == 0
| project DCR, TodayRowsDrop, PriorRows
The Error Table: Where Failures Leave Clues
The metrics tell you that data was dropped, but not why. For that, you need DCRLogErrors, the error table enabled through the Log Errors diagnostic setting.
Once enabled, transformation failures appear here with a message explaining what went wrong. Microsoft recommends alerting on this table with a threshold of one, so any failure triggers an alert.
DCRLogErrors
| where TimeGenerated > ago(1h)
| where OperationName == "Transformation"
| extend DCR = tostring(split(_ResourceId, "/")[-1])
| summarize Errors = count(), Reason = any(Message) by InputStreamId, _ResourceId,DCR
| where Errors > 0
Keep two limitations in mind:
- Repeated errors are logged only a limited number of times per hour, then muted until the hour ends. The limit varies by region, so treat the presence of errors as the signal, not the exact count.
- Some failures cannot be linked to a DCR, such as certain malformed requests and never appear in this table.
It catches a lot, but not everything.
Who Changed What? Check the Activity Log
The queries above tell you that data is being dropped. This one tells you when the rule responsible was last changed.
AzureActivity
| where TimeGenerated > ago(1h)
| where ResourceProviderValue == "MICROSOFT.INSIGHTS"
| where OperationNameValue has "dataCollectionRules"
| where OperationNameValue has_any ("write", "delete")
| extend Action = case(
OperationNameValue has "delete", "Delete",
OperationNameValue has "write", "Write",
"Other"),
DCR = tostring(Properties_d.resource),
ResourceGroup = tostring(Properties_d.resourceGroup),
Subscription = tostring(Properties_d.subscriptionId),
ActorRole = tostring(Authorization_d.evidence.role),
Scope = tostring(Authorization_d.evidence.roleAssignmentScope)
| project TimeGenerated,Action,ActivityStatusValue, DCR,ResourceGroup,Caller,CallerIpAddress,ActorRole, Scope,Subscription,CorrelationId
In a healthy environment, DCRs do not change often. An unexpected write from an unfamiliar caller, happening at the same time as a jump in dropped rows, tells the whole story in one view.
One note: this shows that the rule was changed and who changed it, but it may not show the exact query that was added. If that is the case, treat it as a prompt to compare the current rule with its previous version.
That leads to the strongest control of all.
Control Who Can Change Your Data Pipeline
1.Lock down who can edit DCRs: This is the strongest practical control.
Editing a transformation requires write permission on the DCR, usually through roles such as Monitoring Contributor or Contributor on the resource or resource group. In many tenants, that list is wider than it should be because those roles were granted for other reasons.
Audit who has write access to your DCRs, reduce it to a small named group, and ideally place security-critical DCRs in a resource group with tighter RBAC. The fewer people who can change the filter, the fewer chances it changes without anyone noticing.
2.Require a change ticket: You already alert on DCR writes in the Activity log. Pair that with a simple rule: every DCR change must match an approved change ticket.When the alert fires, check it against the change record. If there is no matching ticket, investigate immediately.
This is a process control, not a tooling control and it works even for changes made directly in the portal.
3.Review existing transformations: Most teams have never read their own transformKql lines.
Export the transformation text for every DCR and review what each one filters. Do this quarterly. It helps catch both a quietly added filter and an accidental rule that drops too much data.
This does not require a pipeline. A recurring calendar reminder and someone responsible for the review are enough.
4.Protect security-critical DCRs more tightly: You do not need to treat every DCR the same way.
For example the rules feeding security tables, such as sign-in logs, device events and audit logs, deserve stricter control than a rule trimming verbose application telemetry.
Identify your security-critical DCRs, apply the tightest RBAC and change-ticket process to them and focus less effort on the rest. Protecting the data that matters most is more realistic than trying to lock down everything.
Your DCR Security Checklist
You do not need to panic. You need five things, starting with the smallest.
- Turn on diagnostic settings for your important DCRs. Send both metrics and the Log Errors category to your workspace. Nothing else works until this is done and it is the step most teams skip.
- Run the baseline query to learn what each rule normally drops. Then turn Signal 1 into a scheduled alert.
- Alert on any new row in DCRLogErrors, so when data is dropped, you also get the reason.
- Alert on DCR writes in the Activity log, so every change to a rule gets reviewed.
- Put your DCRs in control, so changes are reviewed before they reach the pipeline.
Protect the Data Before You Detect the Threat
We spend most of our time improving detections, running hunt and building better correlation rules. But all of this depends on one thing is having the right data.
The pipeline that brings data into Sentinel is often treated as something we set up once and forget. Most teams don’t monitor it as closely as they monitor their logs. But the pipeline decides which events reach Sentinel and which ones get dropped.
An attacker who understands your pipeline doesn’t need to bypass your detection rules. They just need to stop the events from reaching Sentinel in the first place. And it doesn’t always take an attacker. A simple mistake in a transformation can create the same blind spot.
You may not be able to see the events that were dropped, but you can monitor whether data is being dropped, how much is being lost and when the transformation was last changed. Yet many teams don’t monitor these signals.
Take a look at your transformations. You might be surprised by what is being dropped before it ever reaches Sentinel.
Before It Reaches Sentinel: How Your Own Pipeline Can Silently Drop Security Evidence was originally published in Detect FYI on Medium, where people are continuing the conversation by highlighting and responding to this story.
Introduction to Malware Binary Triage (IMBT) Course
Looking to level up your skills? Get 10% off using coupon code: MWNEWS10 for any flavor.
Enroll Now and Save 10%: Coupon Code MWNEWS10
Note: Affiliate link – your enrollment helps support this platform at no extra cost to you.