Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Today I want to break down something that keeps coming up in conversations: the difference between SecOps Agents and AI SOC when they live inside a SIEM versus when they show up as a standalone product or as part of a SOAR.

Three different starting points, three different sets of trade-offs. None of them are wrong. They just optimize for different things, and if you don't understand which one you are buying, you will end up disappointed.

SIEMs own the data, and that matters

SIEMs have the upper hand on triage. Full stop.

If you are a team that sends everything to your SIEM, you will get great triage results inside it. The reason is simple. The SIEM has all the data. If the implementation is done right, the platform can query that data faster and more efficiently than anything bolted on from the outside. Timeline analysis, blast radius, correlation across sources, all the bells and whistles.

But there is a condition attached to this. ONLY if you actually get all the data there. And by data I do not just mean logs. I mean enrichments too. Identity context, asset context, threat intel, business criticality, ownership. The triage is only as good as the context the SIEM can reach without leaving its own walls.

This is where the story gets complicated.

Once you move past triage and into response, the SIEM starts to lose ground. SIEM vendors were smart to play the SOAR card years ago. The problem is that most of them never invested enough to make it shine. Acquired SOAR products got bolted on, kept on life support, and then quietly underfunded while the marketing kept going.

My take has not changed. Unless you have your entire response stack inside a single ecosystem, and very few teams actually do, you need a vendor agnostic agentic, automation, and orchestration layer. The SIEM is a great triage brain. It is rarely a great hands and feet.

Where pure play AI SOC and SOAR have the upper hand

Now flip the scenario. You don't send everything to your SIEM. Maybe you cannot afford to. Maybe you are running multiple detection sources. Maybe your EDR, your cloud detections, your identity alerts, and your email security all live in their own consoles.

This is where pure play AI SOC vendors and SOAR-style platforms shine.

They do enrichment better in this world because they were built for a fragmented data reality. They pull from wherever the data lives instead of assuming it all flows into one lake. The trade-off is that they are limited by whatever APIs are available on the SIEM and on every other system they ingest alerts and detections from. If the API is shallow, the agent is shallow.

On the response side, this is where SOAR-style and agentic platforms really pull ahead. Response is hard. It involves dozens of systems, conditional logic, human approvals, rollback paths, and edge cases that only show up in production. Slapping MCP on top of a tool catalog and calling it response automation does not make the cut. Response needs to be designed, not summoned.

TL;DR for the architecture decision

If you have all your data and all your response actions inside one ecosystem, ride the SIEM agent wave. It will probably work for you.

If you do not, and most teams do not, you need a layer that is vendor agnostic on both data and response. That is where pure play AI SOC and modern SOAR-style platforms earn their keep.

Vendor tracker update

A quick update on the AI SOC and Agentic SOC vendor list we maintain.

We are now tracking 73 vendors. The market keeps expanding, and we keep adding entrants as they show up with credible product and not just a landing page.

A few changes worth calling out:

• We now have multiple visual views of the landscape. Different cuts for different questions. One view for category, one for go-to-market motion, one for how they handle the data and response split discussed above.

• The next iteration will include a definition for each category. One thing I have learned from talking to practitioners and buyers is that the category names are doing a lot of heavy lifting and not always carrying the weight. AI SOC, Agentic SOC, autonomous SOC, alert triage copilot, they all mean different things to different vendors. We will pin down what each one actually means in our taxonomy so the comparison is honest.

If you want to be on the list, or if you think we missed you, reach out. The bar is product that exists and customers who use it.

Coming up: AI SOC is just a feature

Last thing. I will be joining Chris Hughes next week on Resilient Cyber, and one of the topics I want to dig into is why I think AI SOC is just a feature, not a category.

Short version of the argument. Triage automation, alert summarization, investigation assistance, these are capabilities. They will be embedded in SIEMs, in SOAR, in EDR, in detection engineering tools. Standalone AI SOC vendors that do not extend into the rest of the SecOps lifecycle will get squeezed. The interesting companies are the ones building agentic platforms that go beyond the triage box.

More on that in the episode.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

If you've been following this blog, you know the story so far. 2024 gave us AI copilots that summarized alerts but took no action. 2025 proved AI could actually triage, investigate, and reach verdicts at machine speed. The "AI Analyst" became a real product category.

My take: we automated the easy part.

The industry started in the middle of the IR cycle. Triage is analytically complex but operationally simple. No write access required. No change management. No risk of breaking production. It was the perfect first target. Detection engineering sits left of that, tangled in log pipelines and alert volume constraints. Response sits right, blocked by API limitations and organizational risk aversion.

And now we have a new problem.

The Decision Load Tripled

Faster detection and better triage surface more decisions that humans need to make. Before AI, a SOC might process 200 alerts daily and make 50 meaningful decisions. Now AI surfaces 2,000 alerts, auto-closes 1,700, and escalates 300 that require human judgment.

Detection speed improved. Decision velocity did not.

If you read my post on how AI transforms detection engineering, you know I talked about the shift from precision-optimized detection to coverage-optimized detection. The idea is simple: when AI handles triage at scale, you can deploy the detections you always wanted. Broader rules, more coverage, let the AI sort through the noise.

That's true. But it created a side effect nobody planned for.

More coverage means more signals. More signals means more escalations. More escalations means more human decisions. And if those detections are noisy, poorly tuned, or written under the "Fear of Not Doing Enough" (yeah, that post still haunts me), your AI SOC is just processing garbage faster.

Just because AI can investigate and triage alerts faster doesn't mean we should feed it bad detections. We don't want noise detections burning through tokens the same way we didn't want noise detections burning through SIEM licenses. Different cost center, same problem.

Think about it. With traditional SOAR, bad detections cost you analyst hours. With AI SOC, bad detections cost you tokens, compute, and worst of all, they erode trust in the system. Your analysts see the AI confidently closing garbage alerts and start questioning whether it's also confidently closing real threats. That's how you kill adoption.

2026 Will Test the Shift Left

This year will test whether AI can shift left into detection and shift right into response. Not just reorganize the human decision burden but actually reduce it.

I think the shift left is quite important right now. And there are two ways to approach it.

Path 1: The Feedback Loop (Your AI SOC Should Pay for Itself)

If your AI SOC is not giving you suggestions on how to improve your detections, it's designed to charge you more. Full stop.

I've been preaching about feedback loops since the DSAEM days. Detection > SOP > Automation > Emulation > Metrics. The loop is what makes the whole thing work. Without it, you're just driving faster into a wall.

I don't think the AI should give you a one-to-one suggestion for every single alert. That's noise on top of noise. What I see working better is a periodic review. Weekly or monthly. The AI goes over your alerts and incidents, runs analytics across the dataset, and comes back with suggestions.

Things like:

• "This detection fired 847 times last month. 812 were auto-closed as benign. Here's a filter that would eliminate 90% of the noise without reducing true positive coverage."

• "These three detections overlap significantly. You could merge them into one with better context and reduce your alert volume by 30%."

• "This detection has a 2% true positive rate. Either tune it or kill it."

• "Based on analyst overrides, your AI is consistently wrong about this alert type. Here's what the analysts are seeing that the AI is missing."

That last one is gold. When analysts override AI decisions, that's training data. Not just for the AI model, but for your detection engineering program. If the AI keeps getting a specific detection wrong, maybe the detection itself is the problem.

The feedback loop isn't just about making the AI smarter. It's about making your detections smarter. The AI sees patterns across thousands of alerts that no human analyst has time to analyze. Use that.

If your AI SOC vendor doesn't offer this, ask why. Because from where I stand, an AI SOC that doesn't feed back into detection improvement is just an expensive alert processor. And we already had one of those. It was called SOAR.

Path 2: Building Better Detections from Scratch

The feedback loop fixes existing detections. But what about new ones? How do you build better detections from the start?

This is the harder problem. And it's where most teams are still stuck in manual mode.

Let me break down how I think about it.

Where New Detections Come From

In most organizations, new detections come from two places:

Threat Intelligence. You get a feed. IOCs, TTPs, reports about new attack techniques. That intel should feed into your Threat Profile and Threat Modeling. Not every piece of intel is relevant to you. The question is always: does this threat apply to my environment, my industry, my infrastructure?

Threat Hunting. Your hunters go looking for things your detections missed. When they find something, that finding should become a detection. In many cases, threat hunting is also TI-led. The hunter reads a report about a new technique, goes looking for it in the environment, and either confirms or denies its presence.

So the flow looks like this: Threat Intel feeds into both Threat Modeling and Threat Hunting. The outcomes of those processes produce detection suggestions.

Simple enough on paper. In practice? Total mess.

Here's where it falls apart

You get a detection suggestion. Maybe it's a Sigma rule from a community feed. Maybe your hunter wrote it after finding something interesting. The question is: should I implement this?

To answer that, you need to know:

• What log sources do I actually have?

• What infrastructure am I running?

• Do I even have the telemetry to detect this technique?

• If I implement this detection, what coverage does it give me?

• What's the gap if I don't implement it?

Most teams answer these questions from memory. Or they don't answer them at all. They just implement the detection and hope for the best. That's the "Fear of Not Doing Enough" in action. Write the detection, push it to production, deal with the consequences later.

This Is Where AI Should Help

An AI-powered or agentic detection engineering platform should handle exactly this workflow:

1. Ingest your threat intelligence. Not just IOCs. TTPs, actor profiles, campaign reports. Map them to your threat model automatically.

2. Understand your environment. Know what log sources you have, what infrastructure you're running, what telemetry is available. This is the foundation. Without it, everything else is guessing.

3. Match threats to coverage. Based on your threat profile and your available telemetry, show me what I can detect and what I can't. Where are the gaps? What's the risk of those gaps?

4. Suggest detections that make sense. Not generic Sigma rules dumped into a folder. Detections that are relevant to my environment, my log sources, my infrastructure. With the context of what coverage they'll add and what risk they'll reduce.

5. Measure everything. Coverage mapping, risk exposure, detection health. The whole picture.

What to Measure

Since we all love our MITRE ATT&CK bingo cards (don't pretend you don't), coverage mapping is the obvious starting point. But it can't stop there.

Coverage. Map your detections against ATT&CK techniques. Show me what's covered and what's not. Yes, it's bingo. But it's useful bingo when you combine it with the next piece.

Exposure and Risk. For every gap in coverage, I need a way to calculate risk. Not some abstract risk score pulled from thin air. Something grounded in: what threat actors target my industry? What techniques do they use? Do I have compensating controls? What's assumed risk versus mitigated risk?

This is how you go from "we have 60% ATT&CK coverage" to "we have 60% coverage, and the 40% we're missing exposes us to these specific attack paths, with this estimated risk, and here's what we'd need to close those gaps."

That's a conversation a CISO can actually use. Not another bingo card.

Detection Health. Are your existing detections still working? Are they firing? Are they producing true positives? Or have they degraded because the environment changed and nobody updated the rule? This ties back to the feedback loop. A detection you wrote six months ago might be useless today if the infrastructure changed.

Connecting the Dots

If you zoom out, the picture looks like this:

The AI SOC handles triage and investigation. It's fast, it's scalable, it works. But it only works as well as the detections feeding it. Garbage in, AI-powered garbage out.

The feedback loop takes what the AI learns during triage and feeds it back into detection engineering. It's continuous improvement. Your detections get better over time because you're learning from thousands of alert outcomes, not just the handful an analyst has time to review.

Building better detections from scratch takes the proactive approach. Instead of waiting for bad detections to generate noise, you start from threat intelligence and threat modeling. You map coverage, identify gaps, calculate risk, and build detections that actually matter for your specific environment.

The two paths complement each other. One fixes what you have. The other builds what you need.

And this is exactly the shift left that 2026 needs. We proved AI can triage. We proved it can investigate. Now we need to prove it can help us build better defenses from the start.

Vendor Spotlight: Spectrum Security

First platform tackling the agentic detection engineering problem. Details on how they approach the TI > Threat Model > Coverage > Risk workflow, their coverage mapping capabilities, and how they connect threat intelligence to actionable detection suggestions based on your actual environment and log sources.

Spectrum Security is tackling the part of the problem most of the market has worked around for years: detection itself.

While much of the industry focused on collecting more data, shipping more content, or accelerating triage, Spectrum starts from a more fundamental question: can you actually detect the threats that matter in your environment right now? That is the question their platform is built to answer.

The company’s thesis is that security teams do not suffer from a lack of telemetry. They suffer from inability to detect threats through all the telemetry - it’s like searching for a needle in a haystack. More logs do not automatically create coverage. More detections do not automatically reduce exposure. It’s about the need to have the right detections needed for your environment, and ensuring they remain effective as threats evolve and environments change. static dashboards or ATT&CK heat maps do not provide continuous confidence in a changing environment.

Spectrum is building around that gap.

Their approach connects the pieces security teams have historically managed in separate systems and spreadsheets: measuring and reporting on current coverage ,threat intelligence, threat relevance, telemetry availability, detection logic, detection authoring, and ongoing validation. In practice, that means taking threat information and mapping it to the customer’s real environment, understanding what data is actually available, identifying what is detectable versus what is not, and turning that into accurate detections, ensuring that they remain valid and tying this to measurable coverage outcomes. This is meant to replace manually-heavy detection engineering that struggles to keep pace, with something more continuous, contextual, and provable.

What makes the vision interesting is that Spectrum is not describing detection as a one-time engineering project. It is describing it as a living system. Environments change. Telemetry shifts. Threats evolve. Detection logic drifts. So the real problem is not just creating detections, but continuously knowing whether they still work, where gaps have opened, and what matters to fix next. That is the layer Spectrum…

If the first wave of AI in security helped analysts move faster once alerts that already existed, Spectrum is betting the more important shift is upstream: helping teams know what they should detect, what they can detect, and how to close the gap between the two

Final Thoughts

We spent the last two years optimizing the middle of the IR cycle. Investigation and triage are faster than ever. That's good progress. But faster triage doesn't fix bad detections. It just processes them quicker.

The shift left into detection engineering is where the real value is. Not because it's easy. It's actually the hardest part. But because everything downstream depends on it. Your AI SOC, your response automation, your coverage, your risk posture. All of it starts with whether you're detecting the right things in the first place.

Fix the input. The output takes care of itself.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Who Signs Off? AI, The Accountability Anchor and the "Basic" Problem

You have two choices: you either surf the AI wave, or you get caught under it. Surfacing is possible, but the landscape will be unrecognizable.

The current "AI freak-out" isn't actually about Large Language Models (LLMs) becoming sentient. It’s about the displacement of agency. We are moving from a world where humans make every granular call to a delegated model where the Accountability Anchor is the only thing keeping the ship from drifting away. We’ve seen these waves before-from the Luddites smashing looms in 1810 to the SOAR hype that promised the "death of the SOC" a few years ago.

But this time, the wave is massive, powered by a projected $2.52 trillion in global spending by 2026. If we don't build a floor to elevate the workforce, we risk installing a ceiling of obsolescence similar to the "Basic Assistance" trap on Earth in The Expanse series.

The ROI Reality Check: Why the Wave is Rising

Businesses aren't adopting AI because they love the technology; they are doing it because they have to prove value through operational cost reduction and profit maximization. While some companies will take out humans from their jobs just to show a reduction in costs, the smart ones see it as a performance multiplier.

But the threat landscape isn't waiting for anyone to figure out their AI strategy. According to Mandiant's M-Trends 2026 report, based on over 500,000 hours of incident response engagements:

• Speed kills, literally: The median time between an initial access partner breaching an environment and handing off access to a secondary threat group (often ransomware operators) has collapsed to 22 seconds in 2025. Down from over 8 hours in 2022. That's not a typo. Alerts traditionally considered "lower priority" can become full-blown ransomware incidents before a human finishes reading the ticket.

• Dwell time is climbing, not shrinking: Global median dwell time rose to 14 days (from 11 in 2024), driven by espionage operations and DPRK IT worker campaigns averaging 122 days of undetected presence. The idea that we're getting better at detection across the board doesn't hold up when state-sponsored actors are living in your environment for four months.

• Recovery is the new target: Ransomware operators have shifted their primary objective from data theft to deliberate recovery denial, systematically destroying backup infrastructure, identity services, and virtualization management planes. They're not just encrypting your data. They're making sure you can't get it back without paying.

• The spending follows the pain: Average monthly AI budgets are jumping from $62,964 in 2024 to an estimated $85,521 in 2025, a 36% increase. Organizations using AI-powered security platforms report identifying breaches significantly faster and reducing average breach costs by roughly 43%.

When the hand-off from initial access to ransomware deployment happens in 22 seconds, no human is reviewing that alert in time. Defenders are forced to automate just to stay in the game.

The Accountability Anchor: Why Humans Still Own the Risk

In the short to medium term, security jobs are anchored by the legal and ethical need for a human to "sign off." Even as we automate the labor of investigation, the Accountability Anchor ensures that responsibility doesn't vanish into a "black box" where no individual bears the consequences of a failure.

Without human oversight, purely automated decisions can create "accountability vacuums." In high-risk environments like cybersecurity, technology excels at velocity and pattern detection, but it lacks the human capacity for context and consequence.

The Circuit Breaker Problem

The industry has been selling Human-in-the-Loop (HITL) as the safety mechanism that keeps AI in check. The idea is simple: define where the algorithm's authority ends and human discretion begins, force the system to pause at critical moments, hand control to a person who can navigate ambiguity. In theory, this acts as a "circuit breaker."

In practice? The circuit breaker is mostly decorative.

Anthropic recently published data on how users interact with Claude Code permission prompts. The numbers are hard to ignore: 93% of permission prompts get approved, and Anthropic themselves describe this as "approval fatigue, where people stop paying close attention to what they're approving." New users with fewer than 50 sessions auto-approve about 20% of the time. By 750 sessions, that number climbs past 40%.

This isn't unique to coding agents. It's a pattern anyone who has worked in a SOC will recognize instantly. Alert fatigue. Approval fatigue. It's the same cognitive failure mode wearing different clothes. When you ask humans to approve hundreds of actions per day, they stop reading and start clicking.

And the threat landscape is evolving specifically to exploit this gap. M-Trends 2026 shows voice phishing (vishing) jumped to the second most common initial infection vector at 11%, while traditional email phishing dropped to 6%. Attackers aren't sending bulk emails anymore. They're calling people, building rapport in real-time, and exploiting the human tendency to trust a live conversation. The initial infection vectors are getting more human-targeted at the exact moment we're asking humans to be the safety control for AI systems.

The data also reveals something more nuanced than "nobody pays attention." Experienced users don't just approve more; they also interrupt more often. New users review each action upfront and rarely intervene (about 5% of turns). Experienced users let the agent run and step in when something goes wrong (about 9% of turns). This is a deliberate shift from proactive per-action review to reactive monitoring and intervention.

This distinction matters. Per-action approval is not a security control. It's a ritual. The real oversight is happening when experienced operators watch the system's behavior, recognize drift, and pull the emergency brake at the right moment. That is the actual circuit breaker, and it looks nothing like a "click approve" dialog.

When the Anchor Becomes a Rubber Stamp

Here's the uncomfortable question: if the person who is supposed to "sign off" is approving 93% of the time without meaningful review, do you still have accountability? Or do you have compliance theater?

Anthropic's own incident log provides a clear answer. Real-world agentic misbehaviors they've documented include agents deleting remote git branches from vague instructions, uploading an engineer's GitHub authentication token to an internal compute cluster, and attempting migrations against a production database. These are not hypothetical "what if" scenarios. These are things that happened because an agent acted and a human either wasn't watching or clicked "approve" without reading.

The honest answer is that accountability needs to evolve. It can't live at the per-decision approval layer because that layer is broken at scale. The Accountability Anchor needs to move up the stack: the person who answers to the board or the regulator isn't clicking "approve" on every alert closure. They are accountable for ensuring the automation is trustworthy, bounded, and auditable. They own the system design, the guardrails, and the audit trail. Not the individual clicks.

This means building hard boundaries into infrastructure: explicit trust boundaries, tool permissions, action constraints at the architecture level. What environments can the agent access? What actions can it take? What data can it touch? These decisions should be baked into the agent's configuration and enforced programmatically, not left to runtime approval prompts that data shows will get rubber-stamped the vast majority of the time.

M-Trends 2026 reinforces this point from the attacker's side. Ransomware operators are now systematically targeting backup infrastructure, identity services, and virtualization management planes before deploying ransomware. They're not just encrypting your production environment; they're destroying your ability to recover. If your "accountability" layer is a human clicking approve on alert closures, you've already lost. The guardrails need to be baked into the architecture itself: immutable backups, identity isolation, hardened recovery paths. The same principle applies to AI agent governance. Don't rely on the human click. Build the boundaries into the system.

The Responsibility Map

The real gap isn't technology. It's the accountability layer. You can automate the triage, but you cannot automate the person who answers to the board or the regulator when things go wrong. What you can (and should) automate is everything below that person's decision threshold, with guardrails that actually work instead of approval prompts that don't.

ARMM: Evolving Beyond the "Vibe Check"

To move beyond "vibe adoption," we need a maturity model. The AI Response Maturity Model (ARMM), developed by Andrei Cotaie, Cristian Valeriu Miron, and Filip Stojkovski, provides that path.

Maturity isn't just about having an AI; it's about scoring it on three axes:

1. Trust: Do you trust the output enough to let it act?

2. Complexity: Can your team actually maintain this model?

3. Impact: What is the "blast radius" if the AI fails?

What's interesting is that Anthropic's behavioral data validates this progression in the wild. Users naturally move through ARMM levels as they gain experience:

• At Level 2 (AI Assistance), the bot suggests and the human approves every action. This is where new users start, reviewing each step before execution.

• At Level 3 (AI Collaboration), the human shifts to monitoring and intervention. This maps directly to what Anthropic observed: experienced users let the agent run autonomously and interrupt when something drifts.

• At Level 4 (AI Delegation), specialized agents act independently within defined bounds. This is where hard boundaries, deterministic controls, and infrastructure-level guardrails become non-negotiable, because the human is no longer in the per-action loop at all.

The ARMM model helps us evolve roles rather than delete them. But it also forces an honest conversation: if you're claiming to operate at Level 2 while your users are behaviorally operating at Level 3 or 4 (approving everything, monitoring from a distance), you have a maturity gap disguised as a process. Fix the process to match reality, or reality will fix it for you.

The "I, Robot" Shift: When AI Gets a Body

The true long-term impact on the job market isn't just driven by code, but by the "Physical Turn" the convergence of AI and humanoid robotics. This is where the I, Robot vibe becomes a business reality.

• The Scale: Experts project there could be over 1 billion humanoids on Earth by 2050 to offset global labor shortages.

• The Price Tag: While a humanoid cost $200,000 in 2024, costs are expected to drop to $13,000–$20,000 by the early 2030s.

• The Tech: Breakthroughs in "Vision-Language-Action" (VLA) models allow these machines to learn and adapt to unstructured human environments rather than just following a script.

While this sounds like science fiction, it raises the accountability bar to its highest level. A hallucinating chatbot writes a bad email; a hallucinating humanoid has a real-world "blast radius."

The Expanse Metaphor: Floors vs. Ceilings

In the series The Expanse, Earth has a population of 30 billion, but only half have jobs. The rest live on "Basic Assistance" -free food, free housing, and recycled paper clothes, but zero money and zero opportunity.

"Basic" isn't a floor; it's a ceiling. It’s a way to manage a population rendered "obsolete" by automation. As we evolve roles like the Tier 1 SOC analyst, we must ensure we aren't removing the "stepping stones" for new talent. If we automate the path to expertise, we end up with a future of job scarcity where only the "proven" get to work.

We must decide if AI will be used to lift everyone above a "poverty floor" through Universal Basic Income (UBI), or if it will be used to construct a "Basic" ceiling that traps the majority of the population in a state of manufactured scarcity.

Leveling Up: Your Career Anchors

The Tier 1 SOC analyst role isn't disappearing. It's leveling up. The manual grunt work of copying and pasting IPs is being replaced by strategic roles. And the data supports the shift: M-Trends 2026 shows 52% of compromises are now detected internally (up from 43% in 2024), which means organizations investing in detection capability and internal tooling are seeing results. The roles driving that improvement:

• Detection Engineer: Designing the behavior-based models that the AI runs.

• AI Validation Specialist: The person who "validates the autopilot" before the plane takes off.

• Governance Officer: Owning the accountability layer between the silicon and the board.

The three of us all started in the trenches of the SOC. We aren't there now because we evolved with the technology, and the industry is doing the same.

The real question isn't whether AI will take your job-it's whether you'll be the person who owns the risk when it does. But until then, let us thank the heavens for accountability.

We apologize if this felt like a mission briefing for the Rocinante, but Andrei Cotaie is a massive fan of The Expanse and we couldn't stop him from geeking out over the "Basic" problem.

Sources and Further Reading

• Anthropic, "Measuring AI Agent Autonomy in Practice" (2026). The research behind the 93% approval rate, behavioral patterns of experienced vs. new users, and the shift from per-action approval to monitoring-and-intervention.

• Mandiant / Google Threat Intelligence Group, "M-Trends 2026" (2026). Based on 500,000+ hours of incident response engagements in 2025. Source for the 22-second hand-off metric, 14-day global median dwell time, vishing as #2 infection vector, recovery denial trends, and 52% internal detection rate.

• Chris Hughes, "The Human-in-the-Loop Illusion," Resilient Cyber (2026). A complementary analysis of the HITL problem and Auto Mode implications, including Simon Willison's critique on non-deterministic AI safety controls and the UK AISI data on agentic tool growth.
  

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Back from RSAC, still jet lagged. But I need to get this out while the thoughts are fresh.

I have been going to RSAC for years as a practitioner. This year I went as both practitioner and vendor. That changes everything. Running a booth, doing over 20 demos, having 50+ conversations just around the booth. To speak as a vendor I needed to reverse engineer what I was doing as a practitioner, what I wanted to hear, how things needed to be explained so they resonate with everyone. Seeing the conference from both sides gave me a perspective I did not have before, and I think it made this one of the best RSACs I have attended.

More about the vendor experience in a dedicated blog on BlinkOps. Here I want to share from the practitioner lens. What I saw, what got validated, and what the industry is still getting wrong.

The Conference Itself

The quality of talks this year was solid. I have not been on the talk tracks since 2020 when I gave a talk on Intelligent Threat Intel: Lead Framework, so it was good to be back in the sessions.

My favorite was Anton Chuvakin's STR-W08: Shadow Agents: A Pragmatist's Guide to Governing Unsanctioned AI. It triggered some thoughts on how to build better governance around agents. I already drafted a high level design and will release it soon. What made it even better is that some of the things we built at BlinkOps are aligned with how you can actually do better governance for agents and how you can organize them more efficiently.

Beyond the talks, I had a ton of meetings and caught up with a lot of peers. Amazing conversations, great ideas exchanged.

The Uncomfortable Truth About AI Adoption

One theme came up over and over in those conversations. In the past we were able to build tech and say with confidence this is the future and it will be used in the next 3-5 years. Now it is hard to predict how the tech will look in 1 year, all due to the pace of developments around AI and agents.

But here is the interesting part. Not many are at the stage where they implement it at scale. The majority are using AI mainly around copilots. Agentic implementations are still early days and getting traction only between early adopters.

The main issue persists: adoption is messy. It is like we learned nothing from moving from on-prem to the cloud.

A Quick Note on Booth Culture

I will keep this short because I could rant. As usual the floor was full of shiny distracting booths that make you feel like you are in a theme park. If I can't understand what you are selling from a single sentence at the booth, for me that is a no-go. Your booth can have the most awesome visuals and bring cool attractions, but if the messaging is lost then what is the point.

Interesting enough, the smaller booths had better messaging than the large ones. Many of the big ones would just display the vendor name with no context(this works for large well known brands). If I was not stopping to get scanned and watch a demo I had no idea what their product does. That tells me they are not aware of their own brand awareness gap.

AI SOC Is Not a Product, It Is a Feature

This is probably my hottest take from the show, so let me just say it clearly.

AI SOC or Agentic SOC is not a product. It is a feature.

What I saw on the floor is that AI SOC has become a core functionality embedded in many platforms. Almost every SIEM (if not every) now has some form of AI SOC capability. Some are better than others, some are more of a checkbox exercise. But the autonomous triage and base analysis that was initially pitched as a standalone product category is becoming just another feature layer.

On top of that, SIEM vendors all started adding the response (SOAR) piece as well. Elastic for instance announced their Automation capability.

What I heard asked quite often around AI SOC was: How can you give feedback? How does it learn from past experience? Can it be customized? And what else beyond triage can it do?

Those are the right questions. And the vendors that can answer them well will be the ones that survive the consolidation wave.

Where Most AI SOC Vendors Are Falling Short: Response

Now here is where it gets real. The area where I see most AI SOC vendors struggle is response.

Getting a UI where you can build automations is not going to cut it. And doing the lazy route of saying you can connect to MCP and call it a response layer is not going to cut it either. MCP is a protocol not a response strategy.

To do response properly you need a solid integration layer with deep connections into the tools your SOC actually uses. You need orchestration logic, error handling, feedback loops, and many other components that make the difference between a demo and a production deployment. And yes, to be truly functional in this space you need to be able to build agents. Not just use them, build them. That is where the real differentiation lives.

Triage is getting commoditized. Response is where the hard problems are. And most vendors are not there yet.

Predictions Validated

For me RSAC was also a validation moment. Seeing my predictions play out on the floor is a testament that the analysis work we do at SecOps Unpacked holds up.

Prediction 1: AI SOC becomes a feature. Covered above. It is happening across the board.

Prediction 2: AI SOC acquisitions start this year. Not one but two happened already. Culminate was acquired by Datadog. Kenzo Security was acquired by Rapid7. More will come. My bet is we will see at least 5 AI SOC vendors acquired this year.

Prediction 3: AI SOC vendors shift towards MDR or detection engineering. And that is exactly what is happening. The ones that don't go the MDR route are pivoting towards detection engineering. Two clear lanes forming.

Shadow IT and Shadow AI Are Not Going Away

Another topic that persisted through RSAC was Shadow IT and Shadow AI. I think the vendors that will have the most success going forward will be platforms that can govern both shadow IT and shadow AI. The components needed are identities combined with MDM, SSE, ZTNA, and DLP. That is a lot of tech converging in one place, and it probably deserves its own in-depth blog.

Cool Tech Worth Watching

I want to give credit to some teams building impressive things that caught my attention during the show.

Spectrum Security - Finally had a chance to meet with the founding team, Dylan and John Meny. What they are building is super cool and I think it is the kind of tech that could reinvent how we do threat detection.

Above Security - Aviv and his team have some really interesting tech around insider risk. Huge potential there.

Alpha Level - Joshua Neil and his team are building alert management done right. I really like their approach of combining ML and LLM in a smart way. I think this is how you build real IP around AI SOC.

Tracebit - Andy Smith and Sam Cox are building really cool deception technology platfrom.

And yes, for anything else you need BlinkOps. 😀 

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Note: This is Part 1 of two series around AI influence on the SecOps job market

A Brief History of Freaking Out

The Luddites smashed textile machines in the 1810s because they feared losing their livelihoods. Over 200 years later, we're having the same conversation. Not about looms, but about LLMs.

The fear isn't that somebody will replace you in an org chart. It's that the thing you trained for, the thing you're good at, the thing that pays your rent, the thing that is, ultimately, a part of you might become irrelevant.

When we started working in SOCs, a big part of the day was checking indicators across multiple databases. Copy an IP, paste it into five different platforms, check if something shows up, document the results. Repeat. If somebody asked us today whether that job should still exist, we'd all say no. It makes no sense doing that manually when integrations and automations handle it in seconds.

But here's where it gets personal. For us in 2026, losing relevance could mean losing income. The average millennial should expect to change careers five or six times. Some of us are already on that path. But the question is whether we'll be switching by choice or because the tech forced our hand.

This question got a lot more real a few weeks ago when Block (the company behind Square and Cash App) cut 40% of its workforce, over 4,000 people. CEO Jack Dorsey said straight up that "intelligence tools have changed what it means to build and run a company" and predicted most companies would follow within a year. Investors loved it. The stock jumped 22%. For the people who lost their jobs, less exciting.

Now, whether Block's move was truly AI-driven or just pandemic overhiring correction with an AI narrative (there's strong evidence for both), the signal it sent was loud. And it made a lot of people in our industry nervous.

So the question that needs asking: will the jobs that currently define us become irrelevant? Will our industry reinvent itself? Will we survive this wave the same way we survived SOAR's promise that "the SOC is dead"?

We think the answer is more optimistic than the anxiety suggests, but it comes with some conditions.

The Short Answer

Yes, we will survive this wave.

Why? We'll walk through three arguments and stress-test each one. Some of this thinking was shaped by a talk at Apres Cyber Slopes Summit that helped cut through the noise and get back grounded.

Our Industry Is Different

We know, we know. Every industry says this. Every person who ever worked a job says "nobody can automate MY job because of the unknowns I face every day." My job requires intuition. My job requires split-second decisions. A system can't do that.

All of those statements are simultaneously true and false. But security actually has something unique going for it: the offense-defense arms race.

Security was, is, and always will be guided by a simple philosophy: every new defense technology will ignite the spark to create a new weapon.

We're already at the point where both sides use AI. Defenders have AI SOC solutions that automate investigations, speed up detection, ingest new log sources, and discover anomalies. Attackers didn't waste any time either. Every step of the kill chain has been supercharged since LLMs took over. Script kiddies who couldn't do enough harm before are now targeting higher-value assets with better tooling.

Let's take this to the extreme. Say everybody builds a perfect AI SOC that detects everything from day one. What happens the next day? Somebody builds something that manipulates that system into believing their activity isn't worth alerting on. Even in the most automated scenario, somebody will constantly need to detect, train, and alert on new attack patterns. And someone on the other side will keep finding ways to avoid detection.

That cycle doesn't end. It hasn't ended in thousands of years of warfare, and it won't end because we have better chatbots.

Organisational Resistance to Change

The market will force change eventually. But can you imagine the entire population of CISOs saying "yes, please cut 50% of my headcount because AI handles it now"?

Think about what a CISO's leverage is within a company. It's partly about the team they lead. If your headcount drops from 50 to 5, your influence in the executive suite drops with it. No C-level wants that.

And this isn't just about office politics. It's about legal obligation. Which brings us to what we think is the strongest argument in this entire article.

The Compliance Reality Check

This is the argument we think most people are missing.

On one side, fear that AI replaces SOC analysts. On the other side, a reality check. Most security programs are compliance-driven. Many orgs invest in a SOC to pass an audit, not because they love detection engineering.

So what do the compliance frameworks actually require?

We looked at SOC 2, PCI DSS, HIPAA, ISO 27001, NIS2, DORA. Here's what we found:

None of them say humans must do the triage. None of them require a human staring at alerts 24/7. They want continuous monitoring, detection capability, incident response, and evidence. They don't care if the entity doing the work is carbon-based or silicon-based.

Where humans ARE explicitly required: breach notification decisions, risk ownership, audit attestation, governance accountability. NIS2 and DORA put personal liability on executives for security failures.

DORA is the most prescriptive framework out there. Even DORA does not mandate human staffing models. It mandates outcomes.

So the question is not "will AI take SOC jobs." The question is: who signs off that the AI is doing a good job?

That person needs to exist. They need to understand what the automation does. They need to own the risk when it breaks.

When Filip posted this analysis on LinkedIn, Anton Chuvakin called it "quote of the day." And he's right, it IS a big deal. The frameworks already support AI-driven security operations. The real gap isn't technology or regulation. It's the accountability layer between the two.

So What Roles Actually Emerge From This?

The LinkedIn discussion got interesting when Rob Fry jumped in. His point: a lot of people talk about AI as though the vendor ships it, blesses it, and somehow keeps it tuned forever. That's not realistic.

Customer environments, data, workflows, risk tolerance, and operational weirdness are too specific. Vendors can provide the engine, but customers own the care and feeding. Which means you need people who can design, run, validate, and govern AI-driven security systems.

Rob described what he sees as likely new (or evolved) roles:

• Architects to design how AI fits into the control plane

• Operators to monitor, tune, and maintain it in production

• Governance folks to own risk, evidence, and accountability when it fails

• Hybrid roles that sit at the seams between security, engineering, operations, and the business

This lines up with what SACR's recent research on AI SOC and MDR shows. The market is splitting between orgs that can run AI platforms in-house (large enterprises augmenting internal teams) and those that outsource to AI-native MDR providers. Both paths need people. Different people than before, but people.

Will the SOC Analyst Become a QA Role?

Will the SOC analyst role mainly be about QA, just putting the stamp that says "looks good"?

We think partially yes. Someone has to validate the AI's conclusions. But calling it QA undersells what's actually needed. It's more like the person who signs off on the autopilot before the plane takes off. You need to understand the system deeply enough to know when it's working and when it's about to fly into a mountain. The skill set changes. The responsibility doesn't shrink.

And remember how long it took governments to even define what a "data breach" means under GDPR? Now imagine them rewriting accountability laws for fully autonomous security operations. We're not even close to that conversation in most jurisdictions. The current trend is actually toward MORE regulations that require MORE people in different regions, not fewer.

What If We're Wrong?

Fair question. Let's steelman the scary scenarios.

What if the arms race gets solved? Say we get several dozen AI SOC solutions that reach a maturity level where they integrate automatically, train themselves, and run continuous simulations that make any new attack detectable before it becomes real.

This is still sci-fi. It ignores the premise that human innovation and creativity haven't been replaced. The integration work alone is something we can't yet imagine at that scale. And the energy costs would be significant.

What if organizations stop resisting? This one we think is actually the most likely to happen eventually. The market will chip away at resistance once the technology matures. But in such a scenario, it's not just security teams that shrink. Every vertical in an enterprise would be affected, leading to smaller companies across the board.

And that creates a Ford-type dilemma: if nobody is hiring, who buys the products we're making? The "we are building AI for AI" story doesn't solve this economic question.

What if regulations change? Removing rules and regulations requiring human accountability in cybersecurity would be one part of a much larger systemic shift. If you want a historical parallel, look at Khrushchev trying to break down Soviet bureaucracy. Or Gorbachev, who arguably succeeded. Dark humor version: the operation was a success, the patient is dead. The USSR disappeared. Point being, forcefully simplifying complex regulatory systems tends to have consequences way beyond what you planned for.

And remember how long it took governments to define "data breach" under GDPR? If governments become efficient and rational enough to rewrite accountability laws across the board, the entire world would look fundamentally different. The chance of this happening in our industry alone would be a strange mathematical anomaly.

The New Shape of Things

If you've been following the secops-unpacked blog, you know we keep coming back to this: the Tier 1 SOC analyst role as we knew it is disappearing. And it should. It had the worst retention rates, the highest burnout, and it was never a real career destination. It was always a stepping stone.

But the roles replacing it are more interesting. Detection engineering. Security automation. AI validation and governance. These aren't downgrades. They're upgrades.

The three of us all started as SOC analysts. None of us do that job today. We all evolved because the industry evolved. The difference now is that the pace of that evolution is faster. But the pattern is the same: old roles get automated, new roles get created to manage and improve that automation.

Closing Thoughts

The world is changing fast. None of us know what the future holds.

But we do know this: scenarios where our industry faces massive unemployment exist. We just think they're highly unlikely within the next five to ten years.

If we reach a point where cybersecurity jobs are irrelevant in a decade, it means we've had such a massive shift in how society works that job security will be the least of our problems.

For anyone starting their career right now and wondering where to head: the compliance frameworks aren't going away. The accountability layer between AI and business outcomes isn't going away. The arms race between offense and defense isn't going away. Those are your career anchors.

The real question isn't whether AI will take your SOC job. It's whether you'll be the person who signs off that the AI is doing its job right. Position yourself for that, and you'll be fine.

What's your take? Where do you see the roles that AI can't replace in cybersecurity? What arguments do you have to say we won't be unemployed in five years? Drop a comment, we want to hear it.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Alright, here it is. My yearly SecOps trends and predictions report.

Yes, I know it is March. The report was written in January. Life happened. If you want timely content, subscribe to a news outlet. If you want content that ages well, stick around.

Every year I try to make sense of where this industry actually is versus where vendors say it is. Last year I called out the AI copilot hype before everyone got tired of chatbots that summarize things but do nothing. This year I have good news and bad news.

Good news: AI finally started doing real work in 2025. The "AI Analyst" is no longer a slide deck fantasy.

Bad news: We automated the easy part. And somehow created more work for humans in the process.

Let me explain.

Where we stand

MTTD improved dramatically. We got really good at finding things fast. MTTR? Still flat. We are still terrible at actually fixing things.

But here is the part that made me laugh (in a sad way): a new bottleneck emerged. Mean Time to Decision.

Before AI, a SOC processed 200 alerts daily and made maybe 50 meaningful decisions. Now AI surfaces 2,000 alerts, auto-closes 1,700, and escalates 300 requiring human judgment.

We tripled the decision load.

Congratulations. We made the SOC more efficient at generating work for humans.

Why Everyone Started in the Middle

The industry went straight for triage and investigation. Makes sense. It was the easy target. Analytically complex but operationally simple. No write access required. No change management tickets. No risk of breaking production. Just read data, make a verdict, move on.

Detection engineering? Buried in log pipelines, data normalization nightmares, and the eternal fight between coverage and alert volume.

Response? Blocked by API limitations, tribal knowledge nobody documented, and organizations that would rather accept breach risk than give AI systems write access to anything important.

So vendors went for the middle. Quick wins. Happy customers. Logos on the website.

2026 will test whether AI can shift left into detection, shift right into response, and actually reduce the human decision burden rather than just reorganize it.

The SUDA Loop

You know the OODA loop. Observe, Orient, Decide, Act. Military strategy stuff that consultants love to reference.

Security operations needs its own version: See-Understand-Decide-Act (SUDA).

Most 2025 solutions handled one or two stages. See and Understand. Or Understand and Decide. Rarely the full loop.

The platforms that win 2026 will close the entire loop. Point solutions that only serve one stage will get absorbed or left behind.

AI SOC Is Not the Platform

The industry fixated on AI SOC throughout 2025. Understandable. Alert fatigue is painful and visible. Easy to demo. Easy to measure.

But here is my take: AI SOC is one solution. It is not the platform.

The real opportunity is infrastructure that provides building blocks: agentic workflows, deterministic workflows, case management, analyst copilot, integration layer. Combine them to build any security solution your program needs.

Agentic AI SOC. The use case everyone talks about.

Agentic IAM/PAM. Access requests, privilege escalation, orphaned accounts. Identity workflows are still embarrassingly manual in 2026. Let that sink in.

Cloud Security & Vulnerability Management. Findings pile up faster than humans can prioritize. Most sit in dashboards aging like fine wine that nobody drinks.

GRC Automation. Evidence collection, control monitoring, audit prep. The work nobody wants to do, done by systems that do not complain.

Detection Engineering. Threat intel in, detection rules out, coverage gaps identified, feedback loops closed. The dream we have been chasing for years.

Threat Hunting. Continuous hunts based on intelligence and baselines. Not sporadic efforts when someone has time between incidents.

Same platform. Different solutions. Built once, deployed many times.

The Honest Take

Most 2025 AI SOC investment automated the easy part. Organizations declared victory after deploying AI triage, then discovered faster alert closure but the same remediation backlog. Often more decisions queued for human review than before.

Real maturity requires fixing detections, closing the response gap, and making feedback loops actually work. Not just closing alerts faster.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

1. Introduction

The AI SOC market is growing fast and there are products on it that are doing serious work. Some of them have strong integration capabilities, solid reasoning engines, and response actions that actually execute in production. The market has come a long way in four years.

But there is a problem with how we evaluate these products.

When every vendor says "AI-powered response," that phrase covers everything from a fully autonomous isolation workflow to a chatbot that suggests you maybe think about resetting a password. Both get the same label in the marketing material. Both show up in the same analyst reports. And when a security team sits down to compare three products, they have no standardized way to measure the gap between "our AI handles response" and what that actually means in operational terms.

Some products are close to real autonomy in specific domains. Some are strong in analysis but thin on execution. Some have broad coverage but almost nothing runs without human approval. These are all valid positions on a maturity spectrum. The problem is that there is no shared framework to place them on that spectrum consistently.

So we built one.

We call it ARMM. And yes, the name is intentional.

A decade ago, the SOAR generation solved half the problem. We built the arms. Playbooks, integrations, automated response workflows. The execution layer was there. What was missing was the brain. Every decision tree was hand-coded. Every branching logic was written by an engineer who had to anticipate every possible scenario. The arms moved, but only along rails that humans laid down manually. When the scenario deviated from the playbook, the arm froze.

Now the AI SOC generation has solved the other half. We built the brain. LLMs reason across alerts, correlate context, analyze logs, and make judgment calls that no static playbook could replicate. But somewhere along the way, a lot of products forgot to attach the arms. The reasoning is strong. The analysis is sharp. And then it hands you a summary and says "here is what you should probably do." The brain thinks. The arm does not move.

ARMM evaluates both. The reasoning quality, the decision-making maturity, the trust you can place in the AI's judgment. And the response capability, the execution depth, the ability to actually take action without three humans supervising. It weighs the arm heavier because that is where the industry gap is widest right now. But it does not ignore the brain, because an arm without a brain is just a SOAR playbook and we already know how that story ended.

ARMM is a structured scoring system for evaluating what an AI SOC solution can actually do in the response layer. It covers 80+ response capabilities across six domains: Identity, Network, Endpoint, Cloud, SaaS, and General Options. And it provides a common language so that when someone says "we handle response," there is a way to ask: at what level, across how many actions, and with what degree of autonomy?

The CyberSec Automation Blog has published over a dozen articles and podcast episodes covering what makes a good automation program succeed, how to evaluate tools, and how to structure decision-making around security automation purchases. We have built tool comparison lists, evaluation checklists, and decision frameworks. ARMM is the next step in that work.

2. Why Another Framework

Most existing evaluation methods for AI SOC solutions are either vendor-produced (and therefore biased toward their own capabilities) or too generic to capture the specific nuances of AI-driven response. Analyst reports compare products at a feature-list level without measuring automation depth. Vendor demos show best-case scenarios without exposing the operational friction underneath.

Our focus is narrow and deliberate: response capabilities. Most AI SOC solutions already deliver strong reporting and analysis features. They can summarize alerts, correlate indicators, and reduce false negatives in a mature environment (we emphasize mature because these solutions need access to quality logs and, in more advanced implementations, to organizational documentation and environment-specific context). Where the industry needs structured evaluation is in the response layer: the actions an AI SOC solution can take, how autonomously it can take them, and under what conditions.

We acknowledge that some of the capabilities listed in this framework may seem aspirational at this stage. That is by design. The framework is intended to serve both as a current-state evaluation tool and as a forward-looking roadmap.

We are not scoring specific vendors. The goal is to establish a shared methodology that allows security teams to answer questions such as:

• Which solution provides more relevant response capabilities for my environment?

• Which solution operates at a higher level of autonomy for the actions that matter to my program?

• Which solution can help me reduce my alert backlog without requiring additional headcount?

For product managers working on AI SOC products, the framework serves as a competitive analysis baseline:

• Where is my competition positioned, and what capabilities are driving their wins?

• What high-value capabilities are underserved across the market?

• Am I investing engineering resources in features that security practitioners actually prioritize?

Because this is a fast-moving space, we are starting at version 0.1. This is a living document. Version 1.0 will be designated when the framework reaches a level of stability and community validation that warrants it.

3. Scoring Methodology

ARMM supports two distinct approaches to scoring, each designed for a different operational question.

Evaluator Mode is the straightforward path. You score each capability on the 0-1-2 scale described above (with the 1C, 1G, 1A sub-levels) and the framework calculates your coverage rate, automation depth, and per-plane breakdown. The tier placements come from ARMM's reference tables. You do not need to factor in your organizational context. This mode answers one question: given two or more AI SOC products, which one covers more of what I need and at what automation level? It is built for procurement teams, SOC managers running vendor evaluations, and anyone who needs a side-by-side comparison without spending weeks on it.

Builder Mode adds a second scoring layer on top. Instead of relying on fixed reference tiers, you score each action across three axes: Trust (how much confidence does your implementation warrant), Complexity (how hard is it for your specific team to build and maintain), and Impact (what is the blast radius if something goes wrong). The action score becomes T + C + I, and the tier placement shifts based on your organizational reality. The same action that scores Entry for a mature team with established automation pipelines might score Explorer for a team that is deploying its first AI SOC integration. This mode answers a different question: given my team, my environment, and my risk tolerance, where should I invest engineering effort to move up the maturity ladder? It is built for product managers, engineering leads, and internal SOC teams running their own automation programs.

Both modes evaluate the same six planes and the same 80+ response capabilities. Both produce per-plane breakdowns and a composite maturity label. The difference is whether you want a product-level comparison (Evaluator) or an environment-aware implementation roadmap (Builder). The public ARMM app at armm.secops-unpacked.ai  supports both.

3.1 The Capability Scoring System (0-1-2)

Each response capability in the framework is scored on a three-level scale that measures the degree of automation available:

0 (Not Available): The feature does not exist in the product. There is no mechanism, manual or automated, to perform this action through the AI SOC solution.

1 (Available with Human Involvement): The feature exists but requires some form of human interaction before execution. Because human involvement can range from full collaboration to a simple approval click, this level is subdivided into three sub-categories:

• 1C (Collaborator): The solution requires continuous back-and-forth interaction with an analyst to reach a response action. The AI acts as a partner, not an autonomous agent.

• 1G (Guide): The solution generates a plan and presents options for a specific action, but it is not confident in recommending a single path. It lays out alternatives and lets the analyst choose.

• 1A (Approver): The action is essentially ready to execute. The AI has determined the correct response and prepared the action, but requires a human to click approve before it fires. This is the closest step to full automation while still keeping a human in the loop.

2 (Fully Automated): The action is performed without any human involvement. The vendor (or internal implementation) has demonstrated that the AI SOC solution can execute this action with sufficient confidence that no human review is required. At the time of writing, level 2 is exceptionally rare for most response categories. The framework includes it to establish the target state and to differentiate products that are moving in that direction from those that are not.

3.2 The Three Scoring Axes (Builder Mode)

In Builder Mode, each response action is evaluated across three dimensions:

Axis 1: Decision Fidelity and Programmatic Trust (T)

This axis measures the confidence level warranted by the AI SOC implementation. It correlates directly with implementation quality: reasoning log depth, context-aware decision-making, and guardrails against hallucination.

• T = 1 (Enrichment): AI output assists human-led investigations. The AI provides context and data but does not recommend or execute actions.

• T = 2 (Validated): AI recommends a specific action. A human confirms before execution occurs.

• T = 3 (Autonomous): AI executes without human intervention. This requires the highest level of implementation maturity and organizational trust.

Axis 2: Implementation and Maintenance Complexity (C)

This axis evaluates the technical friction in building and sustaining the automation, relative to the skills and resources of the team responsible for it. This is deliberately team-dependent. An automation rated C = 3 for a junior team may be C = 2 for a team of specialized AI engineers with established CI/CD pipelines for their playbooks.

• C = 1 (Low): Simple API calls or native integrations with minimal configuration.

• C = 2 (Medium): Multi-step orchestration across multiple systems requiring coordination and testing.

• C = 3 (High): Complex behavioral baselining, legacy system integration, or custom model tuning.

Axis 3: Operational Impact and Blast Radius (I)

This axis captures the business risk associated with the action. It is typically the most stable axis across organizations, but shifts based on asset criticality. Isolating a standard employee laptop has a different blast radius than isolating a production database server.

• I = 1 (Low): Negligible disruption. Background scans, tagging, enrichment activities.

• I = 2 (Medium): Temporary disruption. Resetting a standard user session, blocking a non-critical port.

• I = 3 (High): Significant downtime, data loss risk, or reputational damage. Production system changes, VIP account modifications, critical infrastructure alterations.

3.3 The Maturity Computation Logic

The scoring system builds from individual actions up to a full program assessment through five layers. Each layer uses a defined formula.

Layer 1: Action-Level Score (S)

For a single response action, the score is the sum of its three axis values:

The minimum possible score is 3 (T=1, C=1, I=1). The maximum is 9 (T=3, C=3, I=3).

Layer 2: Tier Mapping

The action score maps to one of four maturity tiers:

Score Range	Tier	Description
3.00 to 5.99	Explorer	Foundational; low-risk quick wins with minimal blast radius
6.00 to 6.99	Entry	Stabilized; moderate effort and impact, suitable for early-stage programs
7.00 to 7.99	Advanced	Mature; requires high-fidelity reasoning and established trust
8.00 to 9.00	Expert	Critical; high blast radius, autonomous VIP handling, or production-critical actions

Layer 3: Domain Maturity Score (D)

The maturity score for a specific domain (e.g., Endpoint, Identity) is the arithmetic mean of all action scores within that domain:

Where n is the number of scored actions in the domain. The resulting D value maps to a tier using the same thresholds from Layer 2.

Layer 4: Program Maturity Score (P)

The overall program score is the arithmetic mean of all domain scores, with equal weighting across all six planes:

Equal plane weighting is a deliberate design choice. It prevents planes with more actions (Endpoint has 22, SaaS has 10) from dominating the evaluation. Each plane contributes exactly one-sixth of the overall score.

Layer 5: Composite Maturity Label

The composite label is not derived from the program score directly. It uses sequential gating logic:

The composite label equals the highest tier where at least four out of six planes independently meet that tier's threshold, and the qualification chain is unbroken from Explorer upward. A product cannot be labeled Advanced if it has gaps at the Explorer tier.

The four-out-of-six rule is intentionally forgiving. A product focused on cloud-native environments may legitimately deprioritize network-level response. That should not disqualify it from a meaningful composite label. But it still needs breadth across most planes to earn a higher tier.

3.4 Context-Aware Scoring: Why Environment Matters

The ARMM recognizes that the maturity level of an automated action is not a static property of the feature itself. It is an emergent property of the environment where it is applied. The three axes (T, C, I) are all subject to organizational variance, which means the same product capability produces different scores in different contexts.

Example: "Isolate Device" evaluated by three different organizations using the same AI SOC product:

Context	Trust (T)	Complexity (C)	Impact (I)	Score	Tier
Org A: Mature Program / Expert Team	3	2	3	8	Expert
Org B: New Program / Junior Team	1	3	3	7	Advanced
Org C: High-Risk Assets / Manual-First	2	2	3	7	Advanced

The product capability is identical across all three. The scores differ because the Trust axis reflects implementation maturity, the Complexity axis reflects team capability, and the Impact axis (while stable here) can shift based on asset criticality. A vendor benchmark alone is insufficient. Builder Mode exists specifically to capture this variance.

4. Response Capability Domains

The framework organizes response capabilities into six domains. The first five (Identity, Network, Endpoint, Cloud, SaaS) cover specific technical response planes. The sixth (General Options / Usability) covers platform-level characteristics that affect the operational quality of the solution independent of any specific response action.

For the first five domains, each capability is scored using the 0-1-2 system described in Section 3.1 (Evaluator Mode) or the T+C+I system described in Section 3.3 (Builder Mode). For the General Options domain, the scoring criteria shift slightly: 0 means the feature is not available, 1 means the feature is available but limited in capability or partially implemented, and 2 means the feature is fully available, functional, and tested.

4.1 Identity Response Plane

Identity-related response actions target user accounts, service principals, groups, and access permissions. These actions are among the most commonly needed in incident response and are often the first automation candidates for SOC teams.

Action	Description
Reset Password	Reset a standard user's password
Revoke Sessions	Terminate all active sessions for a user account
Disable User	Disable a standard user account
Disable Service Principals	Disable a service account, service principal, or managed identity
Remove Permissions	Remove a specific set of permissions from an account
Group Adherence	Add or remove an account from a security group
Group Creation	Create a new security group
Token Rotation	Create or rotate secrets and tokens
Delete Sharing Permissions	Remove sharing permissions on resources
Label User (Tagging)	Apply a tag or label to a user account for tracking

Builder Mode Reference Scoring (Mature AI SOC Program, Skilled Engineering Team):

Action	T	C	I	Score	Tier
Group Adherence	3	1	1	5	Explorer
Label User (Tagging)	3	1	1	5	Explorer
Revoke Sessions	3	1	1	5	Explorer
Reset Password (Std)	3	1	2	6	Entry
Disable Standard User	3	1	2	6	Entry
Delete Sharing Permissions	2	2	2	6	Entry
Remove Specific Permissions	2	2	3	7	Advanced
Group Creation	3	2	2	7	Advanced
Disable Service Principals	2	2	3	7	Advanced
Reset VIP Password	3	2	3	8	Expert
Rotate Secrets (Prod)	2	3	3	8	Expert

4.2 Network Response Plane

Network-level response actions modify traffic flow, access control, and device connectivity. These are often high-impact actions with significant blast radius, making the Trust and Impact axes particularly important in scoring.

Action	Description
ACL Creation	Create a new access control list on the network
VLAN Creation	Create a new VLAN on the network
Firewall Rule Creation	Create a new firewall rule
IPS Rule Creation	Create a new IPS rule in deny mode
Network Connection Reset	Reset a network connection
DNS Entry Change	Modify an entry in the DNS records
Routing Table Change	Modify a routing entry
Sinkhole Traffic	Redirect traffic to a sinkhole
Rate Limit Traffic	Limit traffic by a particular indicator
VLAN Modification	Move a device to a restricted VLAN
Quarantine Device	Quarantine a device at the network level
Quarantine Server	Quarantine a server running an enterprise-level service
Modify NAT Rules	Change NAT rules to modify traffic patterns

Builder Mode Reference Scoring:

Action	T	C	I	Score	Tier
Network Connection Reset	3	1	1	5	Explorer
Sinkhole Traffic	3	1	1	5	Explorer
Rate Limit Traffic	3	1	1	5	Explorer
ACL Creation	2	2	2	6	Entry
Quarantine Device	3	1	2	6	Entry
Firewall Rule Creation	2	1	3	6	Entry
DNS Entry Change	2	2	2	6	Entry
Modify NAT Rules	2	2	2	6	Entry
IPS Rule Creation	2	2	3	7	Advanced
VLAN Creation	3	2	2	7	Advanced
VLAN Modification	3	2	3	8	Expert
Routing Table Change	3	3	3	9	Expert
Quarantine Server	2	3	3	8	Expert

4.3 Endpoint Response Plane

Endpoint response actions operate directly on devices and their software environment. This domain has the largest number of capabilities because endpoint response spans file operations, process management, application control, forensics, and OS-level changes.

Action	Description
Isolate Device	Isolate a device from all network connectivity
Initiate Malware Scan	Start a scan on the device
Grab File from Device	Upload a file to a designated container
Submit File to Sandbox	Submit a file for sandbox analysis
Lock Out User	Lock a user out of the device
Remove User from Device	Remove a user account from the device
Delete Files	Delete specific files from the device
Kill Processes	Terminate a running process
Remove Application	Uninstall an application
Remove Browser Extension	Remove a browser extension
Modify Browser Settings	Set, modify, or replace browser security parameters
Remove Scheduled Task	Remove a cron entry or scheduled task
Remove Startup Items	Remove a process, agent, or file from system startup
Remove Library / Package	Remove a library from a development environment
Upgrade Application	Force an automatic update on installed software
Upgrade OS	Force an automatic OS update
Deploy Script	Deploy a script or application needed for remediation
Modify Registry Key	Change a value or create a new registry key
Disable Service	Change the status of or remove a service
Collect Memory Dump	Initiate and retrieve a memory dump forensically
Clear Browser Cache	Remove all files, cookies, and data from the browser cache
Remove Device from Domain	Remove a device from the domain

Builder Mode Reference Scoring:

Action	T	C	I	Score	Tier
Initiate Malware Scan	3	1	1	5	Explorer
Clear Browser Cache	3	1	1	5	Explorer
Grab File from Device	3	1	1	5	Explorer
Collect Memory Dump	2	2	1	5	Explorer
Submit File to Sandbox	3	1	1	5	Explorer
Kill Processes	3	1	2	6	Entry
Block File (via Hash)	3	1	2	6	Entry
Lock Out User	2	2	2	6	Entry
Remove Browser Extension	3	1	2	6	Entry
Remove Scheduled Task	2	2	2	6	Entry
Remove Startup Items	2	2	2	6	Entry
Disable Service	2	2	2	6	Entry
Delete Files	2	2	2	6	Entry
Modify Browser Settings	2	2	2	6	Entry
Remove Application	2	2	3	7	Advanced
Remove User from Device	2	2	3	7	Advanced
Remove Library / Package	2	2	3	7	Advanced
Modify Registry Key	2	3	3	8	Expert
Isolate Device	3	2	3	8	Expert
Remove Device from Domain	2	3	3	8	Expert
Upgrade Application	2	3	3	8	Expert
Upgrade OS	2	3	3	8	Expert
Deploy Script	3	3	3	9	Expert

4.4 Cloud Response Plane

Cloud response actions target infrastructure resources, access controls, and storage in cloud environments. The blast radius of cloud actions can be particularly severe because a single misconfigured change can affect multiple dependent services.

Action	Description
Modify Security Group Rules	Modify firewall rules on a cloud resource to restrict access
Create Security Group	Create a new security group and apply it to restrict traffic
Isolate Resource	Quarantine a cloud resource so it is unreachable
Modify Access Type	Switch a resource from public to private or restrict anonymous access
Remove Permissions to Resource	Remove a service principal or managed identity from accessing a resource
Delete Resource	Delete a resource from the cloud environment
Stop Resource	Stop a resource from execution
Modify KeyVault Entries	Add or modify resources in a KeyVault
Use Breakglass Account	Use a breakglass account in case of emergency
Remove Files from Storage	Remove files from a storage bucket or storage account
Copy Storage Device	Create a copy of a cloud storage resource for forensic investigation
Mount Storage Device	Mount a new storage capability to a VM for forensic investigation
Snapshot VM	Create a snapshot of the current state of a virtual machine
Enable Diagnostic Settings	Alter settings that enable advanced log gathering
Apply Resource Lock	Make the resource immutable or read-only

Builder Mode Reference Scoring:

Action	T	C	I	Score	Tier
Enable Diagnostic Settings	3	1	1	5	Explorer
Apply Resource Lock	3	1	1	5	Explorer
Snapshot VM	3	1	1	5	Explorer
Stop Resource	3	1	2	6	Entry
Modify Security Group Rules	2	2	2	6	Entry
Create Security Group	2	2	2	6	Entry
Remove Permissions to Resource	2	2	2	6	Entry
Copy Storage Device	2	2	2	6	Entry
Mount Storage Device	2	2	2	6	Entry
Modify Access Type	2	2	3	7	Advanced
Isolate Resource	3	2	3	8	Expert
Remove Files from Storage	2	2	3	7	Advanced
Modify KeyVault Entries	2	3	3	8	Expert
Use Breakglass Account	2	3	3	8	Expert
Delete Resource	2	3	3	8	Expert

4.5 SaaS Response Plane

SaaS response actions focus primarily on email and productivity platforms, which are among the most common attack surfaces in enterprise environments. Actions in this domain directly affect end-user workflows and communications.

Action	Description
Delete Email	Remove an email from a user's mailbox
Quarantine Email	Move an email to the user's quarantine or junk box
Create Routing Rules	Create rules to handle and route incoming email
Grab Email Sample	Extract an attached file from an email
Grab Email Link	Extract a link from inside an email message
Add / Remove Meeting Invite	Modify a user's calendar
Read / Modify User Status	Read or change a user's status in the HR platform
Disable Malicious Inbox Rule	Disable a rule created by a malicious actor from a user's mailbox
Block Sender	Block a sender from the domain
Modify HR Records	Modify HR records in the system beyond status

Builder Mode Reference Scoring:

Action	T	C	I	Score	Tier
Enable Diagnostic Settings	3	1	1	5	Explorer
Apply Resource Lock	3	1	1	5	Explorer
Snapshot VM	3	1	1	5	Explorer
Stop Resource	3	1	2	6	Entry
Modify Security Group Rules	2	2	2	6	Entry
Create Security Group	2	2	2	6	Entry
Remove Permissions to Resource	2	2	2	6	Entry
Copy Storage Device	2	2	2	6	Entry
Mount Storage Device	2	2	2	6	Entry
Modify Access Type	2	2	3	7	Advanced
Isolate Resource	3	2	3	8	Expert
Remove Files from Storage	2	2	3	7	Advanced
Modify KeyVault Entries	2	3	3	8	Expert
Use Breakglass Account	2	3	3	8	Expert
Delete Resource	2	3	3	8	Expert

4.6 General Options / Usability

This domain evaluates platform-level capabilities that are not tied to any specific response action but directly affect how useful, trustworthy, and manageable the AI SOC solution is in production. The scoring for this domain uses a modified scale: 0 means not available, 1 means available but limited, and 2 means fully available and functional.

This domain is split into two sub-categories to distinguish between operational platform features and AI-specific evaluation criteria.

Platform Operations

Action	Description
Close Alerts in SIEM	The tool can close alerts in all major SIEM solutions
Logging	Platform logging allows identification of all actions taken
Reasoning Logging	Reasoning steps taken by the platform are logged at sufficient detail
API Development	The API is robust enough for integration with other security tools
Support Level	Support is responsive and allows for adequate issue resolution
Account Management	Account management is straightforward with SSO integration
Roles and Responsibility	Role-based access control is available with sufficient granularity
Ease of Use (GUI)	The GUI is navigable and intuitive
Native Chat Integration	Native integration with major communication platforms
Alerting	Automatic alerting when platform-level or analysis-level issues arise
Stats / Health Dashboards	Dashboards showing current platform status and performance

AI-Specific Evaluation Criteria

Action	Description
Bring Your Own Model	Ability to integrate custom models into the platform
Context Grounding	Ability to bring organizational data to feed into the ML model
Autonomous Action Thresholds	Platform allows setting confidence thresholds for autonomous execution
Investigation Audit Trail	Complete, exportable record of every action (AI and human) with timestamps
IR Metrics Tracking	Native tracking of MTTD, MTTA, MTTT, MTTI, MTTR without external tooling
Feedback Loop Mechanism	Analysts can confirm, reject, or correct AI decisions with feedback incorporated
Auto-Close Reversal Tracking	Tracks the rate at which auto-closed alerts are reopened by analysts
Explainability / Decision Transparency	AI provides clear, traceable reasoning for every decision
AI Decision Accuracy Reporting	Tracks TP accuracy, FP accuracy, and confidence scores over time
Model Drift Detection	Monitors AI model performance and alerts when accuracy degrades
Adversarial Robustness Testing	Supports or integrates with red team exercises to test AI resilience

5. Aggregate Maturity Scoring

Evaluating each plane individually is necessary but not sufficient. Security teams making purchasing decisions and product managers tracking competitive positioning need a consolidated view that communicates the overall picture without hiding the details.

5.1 Automation Depth Score

This is the most operationally significant metric and the one that separates real autonomous solutions from products that wrapped a chatbot interface around a set of API calls.

Across all covered capabilities, calculate the distribution:

• What percentage is fully automated (level 2)?

• What percentage sits at Approver level (1A)?

• What percentage sits at Guide level (1G)?

• What percentage sits at Collaborator level (1C)?

• What percentage is not available at all (0)?

A product could have 80% of capabilities covered but only 5% fully automated. That is a fundamentally different product than one with 60% covered but 40% fully automated. The first is broad but shallow. The second is narrower but operates with real autonomy where it counts.

Full Automation Rate: The percentage of total capabilities at level 2. This is the true measure of how much an AI SOC solution can operate without human intervention.

Coverage Rate: The percentage of total capabilities at any level above 0. This measures breadth regardless of automation depth.

The relationship between these two numbers tells you everything about how the product actually operates. A high coverage rate with a low automation rate means the product is a guided workflow tool with AI branding. A moderate coverage rate with a high automation rate relative to coverage means the product is autonomous in its areas of focus but limited in scope.

5.2 Combined Scoring Readout

A complete ARMM evaluation for a product produces the following consolidated output:

Metric	Value
Overall Score	47% (equal plane weighted)
Composite Maturity	Entry (5 of 6 planes at Entry or above)
Automation Depth	12% fully automated, 61% covered at any level

Per-Plane Breakdown:

Plane	Score	Coverage	Fully Automated	Tier
Identity	78%	7/9 covered	2 actions	Advanced
Network	42%	8/13 covered	1 action	Entry
Endpoint	38%	12/21 covered	1 action	Entry
Cloud	31%	6/15 covered	0 actions	Explorer
SaaS	55%	7/10 covered	3 actions	Entry
General Options	64%	10/14 covered	3 actions	Advanced

6. Reading the Model

A product can reach Expert level on a specific plane by checking all the boxes for that domain. But it would be difficult to consider an AI SOC Response solution as Expert level overall if it lacks the ability to perform foundational actions like closing alerts in a SIEM. The tier system is designed to reward both depth within a domain and breadth across domains.

The reference maturity tables provided in Section 4 use example scores from a hypothetical mature AI SOC program with a skilled engineering team. These are illustrative, not universal benchmarks. The environmental dynamics described in Section 3.5 are not optional context; they are a core part of how the framework is intended to be used.

When comparing two products, the most informative comparison is not the aggregate score. It is the per-plane breakdown combined with the Automation Depth Score. Two products at the same composite tier can have radically different operational profiles. One may cover 80% of capabilities at the Collaborator level. The other may cover 50% but with 30% at full automation. These are different products for different buyers with different operational maturity levels.

7. Limitations and Future Work

This is version 0.1. The framework has known limitations:

• The capability lists are not exhaustive. New response actions will emerge as AI SOC products mature and as attack surfaces expand.

• The three-axis scoring (T, C, I) requires subjective judgment that will vary between evaluators. We plan to develop calibration guidelines to reduce inter-evaluator variance.

• The framework does not currently weight domains differently. In practice, Identity response may be more important than Network response for a given organization. Weighted scoring is planned for a future version.

• Detection and analysis capabilities are out of scope for this version. A separate framework or an extension to ARMM may address those in the future.

• We have not included pricing, deployment time, or vendor lock-in considerations. These are important purchase factors but are outside the scope of a technical maturity model.

We are building a public web application where users can input their product's capabilities and generate ARMM scoring layers automatically, along with an exportable CSV. The application is available at: armm.secops-unpacked.ai

8. Conclusion

The AI SOC market is growing faster than the industry's ability to evaluate products on consistent terms. The ARMM framework provides a structured, repeatable methodology for measuring what an AI SOC solution can actually do in the response layer, how autonomously it can do it, and what it takes to deploy and maintain that capability in a specific operational environment.

The framework is built for two audiences: security teams evaluating products and product managers building them. For security teams, it provides a checklist and scoring system that cuts through marketing language and focuses on operational capability. For product teams, it provides a competitive analysis baseline and a prioritization framework for feature development.

SOAR gave us arms without brains. The first wave of AI SOC products gave us brains without arms. The products that will win this market are the ones that connect both. ARMM gives you a way to measure how far along that connection is, and where the gaps remain.

No current AI SOC solution will check every box. That is not the point. The point is to establish a common language and a common measurement system so that the conversation about AI SOC response capability is grounded in specifics rather than promises. Version 0.1 is the starting point. The framework will evolve as the market does.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

The Fear of Not Doing Enough: Security's Workflow Problem

If you've been following this blog, you know I've spent a lot of time on AI transforming investigation, triage, and detection engineering. And a few months back I wrote about the single pane of glass, how it's not a product you buy but a system you build, piece by piece, like Legos.

That post was about the architecture. What tools do you need, how do they connect, where does data flow.

This post is about the layer underneath that nobody talks about. Not the tools. The work itself. Where it comes from, how it flows, and why we have zero visibility into most of it.

The Fear of Not Doing Enough

Security has this pattern I keep seeing everywhere.

New attack technique drops. A CVE trends on Twitter. Some threat intel report lands in your inbox with a fancy APT name. What happens next? Predictable.

Someone writes a generic detection rule fast so the team has "something." Gets pushed to production. Generates noise. Nobody tunes it because there's already another thing screaming for attention.

The false sense of coverage becomes more important than actual coverage.

I call this the Fear of Not Doing Enough. And honestly? It drives most of the operational pain in security teams today.

You write a detection rule. Do you have the SOP for when it fires? Do you know the full analysis path an analyst should follow? Can you estimate how that alert impacts your team's workload downstream? Do you know what "done" looks like for that alert type?

If you can't answer those, you didn't deploy a detection. You deployed a work generator with no operating manual. Multiply that across dozens of detections written under pressure and you get patchwork coverage that looks great on a dashboard but falls apart when someone has to actually operate it.

But here's the thing. Even if you fix all of that, you're still only looking at one input stream.

It's Not Just SIEM Alerts

A ACM Computing Surveys paper (Tariq et al., 2025) reviewed over 30 solutions to alert fatigue in SOCs. Thorough paper, I'll give them that. Identifies four root causes: staff shortage, high false positive rates, disconnected dashboards, and inefficient SOPs.

But every single solution assumes the work starts with a SIEM alert.

Now look, I'm not saying SIEM alerts are a small part of the work. For most teams they're probably more than half. But here's what matters: the work that doesn't come from the SIEM is often the most manual, least structured, and hardest to track.

IT escalations. Someone from the help desk pings you on Slack: "Hey, this looks weird." Access review requests from HR. Audit findings that need remediation tracking. Pen test findings that need to be assigned and fixed. Third-party risk questionnaires. Compliance asks from legal.

All real security work. And here's the thing about it: only some or none of it has a playbook or automation behind it. Most of it lives in Slack threads, email chains, and spreadsheets. It's the security work that runs entirely on copy-paste, tribal knowledge, and good intentions.

Your SIEM alerts, for all their problems, at least flow through a pipeline. They get enriched. They have some structure. Maybe even a SOAR playbook attached. The non-SIEM work? It's the Wild West.

Erik Bloch has been making this point for years.A lot of the work SOC is doing day-to-day has nothing to do with chasing advanced adversaries. It's tickets, reports, evidence collection, reconciling data across tools. The mundane operational grind that actually burns people out.

And here's the part that really gets me. Outside of very large enterprises that have 10 security sub-departments with dedicated teams for everything, the same 3-5 people triaging SIEM alerts are also pulling evidence for the auditor, handling the IT escalation, and answering the compliance questionnaire. There's no luxury of specialization. The alert queue is just one input stream among many. And the non-SIEM stuff eats time disproportionately because it's all manual.

Security Work Has No Gravity

Ross Haleliuk recently wrote a great piece about ServiceNow betting on "workflow gravity" to compete with the security platform giants. The thesis is simple. Whoever owns where work happens owns the decisions.

Data gravity pulls information into a single system of record. Your SIEM, your data lake, whatever. That part most teams have figured out. Workflow gravity is different. It pulls action into a single system of action. One place where work lands, gets triaged, gets tracked, and gets done.

Right now? Security work has no gravity. It's everywhere and nowhere.

And yeah, this connects directly to the single pane of glass conversation. In that post I talked about building your own platform, Lego-style, with assets, data layers, correlation, and response actions. But even if you build that beautiful architecture, it's still oriented around machine-generated alerts. The SIEM brain, the enrichment layer, the correlation engine. All of that assumes the input is a structured alert.

What about the IT manager who emails you about a suspicious contractor? What about the audit finding that needs 6 teams to remediate? What about the pen test report sitting in a shared drive that nobody has turned into action items yet?

That work has no architecture. It has no pipeline. It just shows up and someone deals with it however they can.

You want to know why security teams always feel understaffed? Part of it is real headcount shortage, sure. But part of it is that nobody can actually see where the time goes. When the most manual, time-consuming work lives outside of every system you've built, you can't measure it. When you can't measure it, you can't optimize it. When you can't optimize it, you just throw more people at it and hope for the best.

Process Mining Exists. Just Not for Us. Yet.

Here's something that gets me. In finance, procurement, and operations, tools like Celonis and Scribe Optimize have existed for years. They observe how work actually happens across tools and systems. They find bottlenecks. They tell you where time is wasted. They optimize based on data, not vibes and assumptions.

In security? Still very early days.

Some vendors are starting to take RPA-style approaches to There's a handful of academic papers exploring it. But it's nowhere near mainstream.

We still don't have good data on how security work actually flows end to end. Think about that.

We have terabytes of security telemetry. We can tell you exactly when a process spawned on an endpoint at 3:47am. But we can't tell you how long it takes an analyst to go from "alert fired" to "investigation complete." We can't tell you how much time the team spends on compliance requests versus actual threat work. We can't tell you which of your 200 detection rules generates the most operational overhead relative to the security value it provides.

That's wild.

Why This Is Hard

I get why the industry keeps gravitating toward the easier wins. Make investigation faster. Automate the playbook. Build a better ML model for triage. Those are well-defined problems with measurable outcomes.

Understanding where all security work happens and how it flows? That's messy. It crosses tool boundaries. It involves human behavior that doesn't fit neatly into event logs. It requires looking at the whole system, not just one piece.

This is the hardest problem to solve. And that's exactly why not many are tackling it yet.

But here's why it matters. If you don't understand the full picture of how work enters and flows through your security team, everything else you build is an optimization of a subsystem. You can make SIEM triage 10x faster, but if a third of the work comes from non-SIEM sources that are entirely manual, you just made one part of the problem better while the messiest part stays untouched.

What Would Actually Help

I don't think this needs to be one giant platform that replaces everything. But teams need a few things that barely exist today.

Workflow data. How long does each type of work actually take? Where are the handoffs? Where do things stall? What percentage of the team's time goes to which category of work? Right now most teams are guessing. And the guesses are usually wrong because the most painful work is the least visible.

Operational impact awareness. Before you deploy a new detection, onboard a new data source, or agree to a new compliance requirement, you should be able to model what that does to your team's capacity. Not after the fact when everyone's drowning. Before.

Connection between detection and process. If you have a detection but you don't have the analysis path mapped from it, you can't estimate how it impacts anything downstream. Every detection should ship with its SOP. Not as a nice-to-have. As a requirement.

The Fear Won't Go Away

The Fear of Not Doing Enough will always be there. New threats aren't going to stop coming. The pressure to have "something" for every new attack vector is real.

But the answer isn't to keep throwing generic detections at every new thing and hoping the team can absorb the blast. It's not to keep building faster investigation tools for one slice of the work while the rest drowns in Slack threads and spreadsheets.

We've been fixing the middle. Investigation is getting faster. AI triage is real. Response automation is improving. The single pane of glass architecture is getting clearer. All good progress.

Now it's time to zoom out. Understand how security work actually flows. All of it. Not just the structured, machine-generated part. Especially the messy, manual, human-generated part that eats the most time and has the least tooling.

Fix the input. Model the cost. Understand the workflow.

Stop optimizing the output of a system you've never fully mapped.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Hey everyone. Been quiet on here for a bit. First post of 2026.

I came across a paper that finally articulates something I've been thinking about for a while: autonomy isn't a capability score. It's a design decision"Levels of Autonomy for AI Agents". And chasing the highest level everywhere is a mistake.

Let me walk through it.

L1 (Operator): User directs and makes decisions, agent acts. Think Microsoft Copilot. The agent requires invocation to act, provides on-demand assistance, and avoids preference-based decision-making on the user's behalf.

L2 (Collaborator): User and agent collaboratively plan, delegate, and execute. Think OpenAI Operator. Users can freely modify agent work and take control at any point. Back-and-forth communication is frequent.

L3 (Consultant): Agent takes the lead but consults user for expertise and preferences. Think Gemini Deep Research. Users provide feedback and directional guidance rather than hands-on collaboration. The agent bears more of the learning curve.

L4 (Approver): Agent engages user only in risky or pre-specified scenarios. Think Devin. Users specify approval conditions upfront. The agent only stops for blockers, credentials, or consequential actions.

L5 (Observer): Agent operates with full autonomy under user monitoring. Users can watch activity logs and hit the emergency stop. That's it.

The key insight: autonomy is a design decision, not a capability metric. A capable agent can still operate at L2 if that's the right call for the task. The paper explicitly argues against treating autonomy as an inevitable consequence of increasing capability.

Agency vs. Autonomy

The paper makes an important distinction that matters for security operations.

Agency is the capacity to carry out intentional actions. It's about what tools the agent has access to and what it can do in the environment.

Autonomy is the extent to which the agent operates without user involvement. It's about when and how the agent checks in with humans.

An agent with high agency (many tools, broad permissions) can still have low autonomy (checks in frequently). An agent with low agency (limited toolset) can have high autonomy (runs independently within that scope).

This distinction matters because security teams often conflate the two. Giving an agent access to more data sources (agency) is different from letting it act without approval (autonomy). You can expand agency while constraining autonomy.

Mapping Autonomy to Security Operations

I mapped common security workflows to appropriate autonomy levels based on their risk profile and decision complexity.

Initial Triage: L4-L5

For alert triage, scale beats precision. You're dealing with volume. The goal is filtering, not final judgment.

L4 makes sense here. Let agents do the heavy lifting, have them seek approval only for edge cases or high-severity alerts. L5 is reasonable for low-fidelity alerts where false positives cost nothing. Start at L4. Keep humans reviewing outcomes for anything that escalates to investigation.

The paper notes that L4 agents are ideal for tasks with high amounts of lower-stakes decision-making. Alert triage fits this description. Automated decisions improve efficiency. Erroneous decisions on individual alerts don't impose catastrophic risks if the escalation path is intact.

Incident Response: L1-L2

On the response side of the IR cycle, L1 and L2 work well. These are established patterns with runbooks and playbooks.

Why keep autonomy low here? Response actions are consequential. Isolating a host, blocking a domain, killing a process. These actions have real impact. Speed matters, but accountability matters more.

L1 is appropriate when analysts drive the workflow and agents execute specific tasks on command. L2 works when you want the agent to propose containment plans while the analyst retains takeover capability.

The paper describes L2 control mechanisms as requiring "control transfer from agent to user, and vice versa" plus "shared representation of progress." That maps well to incident response dashboards where analysts can see what the agent is doing and intervene.

In-Depth Investigation: L3

Deep investigation is judgment-heavy work. Context matters. The analyst brings domain knowledge, institutional memory, and threat intelligence the agent doesn't have.

L3 fits this workflow. The agent leads the investigation, gathering data, correlating events, building timelines. But it consults the analyst for direction. What's the hypothesis? Which threads are worth pulling? Does this pattern match something we've seen before?

The paper notes that L3 agents require "productive and timely consultation." The agent needs to know what expertise the user brings and when to ask for it. For security investigations, this means the agent should surface findings and ask about relevance rather than drawing conclusions autonomously.

Threat Hunting: L2-L3

Hunting is exploratory by nature. You're looking for things you don't know exist yet. Hypotheses matter. Intuition matters.

Collaboration beats full automation here. L2-L3 is the range. The agent surfaces anomalies, suggests investigation paths, runs queries. The human drives the hunt itself.

The paper describes L2 as the level where "back-and-forth communication between the user and the agent is the most frequent and rich." Threat hunting benefits from this dynamic. The hunter's domain expertise combined with the agent's ability to process large datasets creates a feedback loop that pure automation can't replicate.

Detection Engineering: L2-L3

Detection engineering is systematic but consequential. Bad detections create alert fatigue. Missed detections create gaps.

L2 is the baseline. The agent assists with query building, suggests detection patterns, helps test against historical data. The engineer retains control over what gets deployed.

L3 is appropriate for mature teams with well-governed detection lifecycles. The agent drafts detections, runs validation, and consults the engineer before deployment. The key is having proper testing and review controls already in place.

The paper warns about L4 agents and "meaningless rubber stamping" from user disengagement. This risk is real for detection engineering. If engineers just approve whatever the agent proposes, detection quality will degrade.

The Double-Edged Sword

The paper repeatedly emphasizes that autonomy amplifies both benefits and risks. Higher autonomy means more scale and efficiency. It also means errors compound over multiple steps without intervention.

This maps directly to security operations. An agent that autonomously closes false positive alerts at L5 saves analyst time. An agent that autonomously closes true positive alerts at L5 creates security incidents.

The paper also raises concerns about deskilling and loss of critical thinking when automation takes over judgment tasks. Security teams should consider this. If agents handle all investigation, what happens to analyst skill development? L2 and L3 autonomy levels preserve opportunities for human engagement while still providing automation benefits.

Autonomy Certificates

The paper proposes "autonomy certificates" as a governance mechanism. A third-party body evaluates an agent's behavior and certifies the maximum autonomy level at which it can operate.

This concept has implications for security vendors. Right now, every AI SOC vendor claims some version of autonomous operation. There's no standard way to compare what that actually means.

An autonomy certificate framework would force clarity. Does your agent operate at L3 or L4? What approval mechanisms exist? Under what conditions does it escalate?

For security buyers, this creates better evaluation criteria than vague claims about AI capabilities.

Double-Layer Governance: Reasoning and Abilities

The agency vs. autonomy distinction from the paper points to a practical governance model. You need to control both what the agent can think about doing and what it can actually do.

At BlinkOps, we implement this as double-layer governance:

Layer 1: Reasoning Constraints

This layer limits what the agent can decide to do. It's autonomy governance. You define the scope of problems the agent is allowed to reason about and the types of conclusions it can reach.

For example, an agent handling alert triage might be constrained to reason only about severity classification and enrichment. It can't decide to initiate response actions, even if it has the technical capability. The reasoning boundary is set before the agent ever considers what actions to take.

This maps to the paper's definition of autonomy as "the extent to which an AI agent is designed to operate without user involvement." By constraining reasoning scope, you limit how far the agent goes before involving a human.

Layer 2: Ability Constraints

This layer limits what the agent can execute. It's agency governance. Even if the agent reasons its way to a valid conclusion, it can only act through explicitly permitted capabilities.

This is your tool allowlist. The agent might determine that isolating a host is the right response, but if host isolation isn't in its permitted action set, it can't execute. It has to escalate.

This maps to the paper's definition of agency as "the capacity to carry out intentional actions." By constraining the toolset, you bound the blast radius of any autonomous decision.

Why Both Layers Matter

Single-layer governance creates gaps.

If you only constrain abilities (Layer 2), the agent can still reason about actions outside its scope and make recommendations that push humans toward decisions the agent shouldn't influence. An agent without response permissions might still conclude "this host should be isolated immediately" and create pressure for hasty action.

If you only constrain reasoning (Layer 1), the agent might find edge cases where its reasoning scope overlaps with dangerous capabilities. A triage agent reasoning about "enrichment" might decide that querying a production database for context falls within scope.

Double-layer governance closes both gaps. The reasoning layer defines intent boundaries. The ability layer enforces execution boundaries. An action only happens if it passes both checks.

Practical Implementation

For each workflow, define:

1. Reasoning scope: What questions can the agent answer? What conclusions can it reach? What types of decisions are out of bounds?

2. Action permissions: What tools and integrations can the agent invoke? What parameters can it set? What requires human approval?

3. Escalation triggers: When reasoning hits scope boundaries, where does it go? When actions require approval, who approves?

This gives you granular control without blocking automation entirely. An L4 agent can still operate autonomously within its defined scope. But that scope is explicitly bounded at both the reasoning and execution layers.

The paper's framework helps here. L4 requires "customizable conditions for seeking approval." Double-layer governance operationalizes this. The conditions are defined by reasoning scope violations (Layer 1) and action permission requirements (Layer 2).

What This Means for AI SOC Design

If you're building or buying AI-powered security tooling, ask different questions:

What autonomy level does this workflow need? Not "how autonomous is this agent?" Match the autonomy to the task risk profile.

What are the must-have controls? Each autonomy level has required control mechanisms. L4 requires approval elicitation for consequential actions and customizable conditions. L2 requires control transfer mechanisms and shared progress visibility. Verify these exist.

Where are the approval gates? Every workflow should have defined checkpoints. Know what triggers human involvement.

What's the fallback? When the agent hits a failure state, what happens? The paper notes that L4 and L5 agents should iterate on solutions or modify approaches when blocked. How does your agent handle this?

Who's accountable? Higher autonomy means harder accountability tracing. The paper cites research showing it's simultaneously more important and more difficult to anticipate harms from autonomous AI. Design governance around this reality.

Closing Thoughts

Chasing L5 everywhere is a design mistake, not a strategy.

The vendors pushing "fully autonomous SOC" are selling a destination most teams shouldn't want to reach. The right autonomy level varies by task, by maturity, by risk tolerance.

The paper's framework gives us a shared vocabulary for these discussions. Use it.

Reference: Feng, K.J.K., McDonald, D.W., & Zhang, A.X. (2025). Levels of Autonomy for AI Agents. University of Washington. arXiv:2506.12469

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Article written in collaboration with Andrew Green

We often argue over semantics, but your agents shouldn't. They need explicit definitions for what data and events mean for your business. Otherwise they will refer back to their training data, which is often not applicable.

For example, a log from an on-prem Cisco router and an AWS VPC log are structured differently, but they both contain IP addresses, ports, protocols. An LLM can understand what these network elements mean and make general inferences about network requests. But how will it determine which instances of traffic between your on-prem environment and cloud are expected versus suspicious?

You've got three options here.

The first one is to ask the LLM: 'Hey, is this suspicious?', at which point the LLM will be using its training data and prompt context to give a statistically likely interpretation. This is the neural approach.

The second one is to define an explicit rule, such as alert if request.source_ip in "10.0.0.0/8" and request.destination_ip in "52.0.0.0/8". This is the symbolic approach.

The third one is to tell the LLM: 'Hey, our dev Jerry had a leg surgery and he won't leave the house while he heals, but he'll be working from home'. With the right tools, the LLM will now interpret the ambiguities of this message and combine them with hard rules to define an explicit instruction, where all of Jerry's access requests will be made from his VPN for the next 6 months, while anything else is suspicious. This is the neurosymbolic approach.

To get to this third option, we need a semantic layer.

The Fundamental Tension: Why We Need Semantic Layers

Before we define what semantic layers are, let's understand the problem they're solving. There's a fundamental tension in how we approach automated reasoning in security:

Pure Neural (LLM) Approach:

• Excellent at pattern recognition and handling ambiguity

• Can process natural language queries

• Adaptable to new situations

• BUT: Non-deterministic, expensive at scale, can't explain decisions

Pure Symbolic Approach:

• Deterministic and auditable

• Fast and efficient at scale

• Provides clear reasoning chains

• BUT: Brittle with edge cases, requires extensive rule maintenance

The semantic layer provides a foundation for hybrid approaches where symbolic reasoning handles the facts and constraints while neural networks handle the ambiguity and interface. It's the bridge between "Jerry is working from home while recovering" (natural language) and "source_ip = Jerry_VPN_IP AND time_range = next_6_months" (executable logic).

But What Is a Semantic Layer, Exactly?

A semantic layer is an LLM-queriable abstraction layer that pulls and correlates information about entities, their relationships, and the deterministic rules that govern their interactions. 

It correlates raw technical data with business meaning through symbolic reasoning.

But what is symbolic reasoning?

Actually, what is a symbol?

A symbol is a notation that captures the meaning of a concept, such as 'putting two things together' is expressed by the symbol '+'. Humans and LLMs can use these symbols to read and define rules and instructions.

Symbolic reasoning therefore uses these notations to work out a conclusion.

At its core, a semantic layer must define the following three concepts:

1. Entities: identities (both human and workload), assets such as servers and applications, events like a GET request, and behaviors such as Dev (identity) makes a POST request (event) to a database (asset).
   

2. Relationships also known as relationship graphs or knowledge graphs, these map things like User X manages Asset Y, Asset Y hosts Application Z, Application Z processes Data Type W, Data Type W is subject to Regulation V.
   

3. Symbolic Reasoning: applying logic-based rules. You can trace exactly why a decision was made. Every step is auditable, testable, and deterministic.
   
   a. IF user.role = "contractor"
   
   b. AND access.time NOT IN business_hours
   
   c. AND asset.classification = "confidential"
   
   d. THEN risk_score = HIGH
   

Where Should Semantic Layers Live?

Considering most security activities revolve around gathering accurate and relevant data, where in the modern SecOps stack should this semantic layer be inserted?This is where theory meets painful reality, and where practitioners diverge sharply on the right approach.

There are four main architectural positions, each with compelling arguments and serious tradeoffs:

Option 1: In the Data Pipeline (Pre-SIEM)

The argument here is to do semantic processing before data ever reaches your SIEM or data lake.

Advantages:

• Reduces data volume through intelligent filtering

• Normalizes entities at the source

• Cheaper than processing in expensive SIEM platforms

• Can route data based on semantic understanding

Challenges:

• Requires real-time processing at massive scale

• Changes to semantic models require pipeline updates

• Limited context (can't look at historical data easily)

• Becomes another system to maintain

Verdict: This works well for basic entity resolution and enrichment, but struggles with complex relationship modeling that requires historical context.

Option 2: Within the SIEM Platform

Modern SIEMs increasingly claim to embed semantic capabilities directly.

Advantages:

• Integrated with detection and investigation workflows

• Access to all historical data for context

• Single platform to manage

• Vendor support (in theory)

Challenges:

• Vendor lock-in to proprietary semantic models

• Performance impacts on already-stressed SIEMs

• Limited flexibility in modeling

• Expensive processing at SIEM rates

Verdict: most SIEM vendors are adding "semantic" features, but they're often just rebranded lookup tables and data models. True semantic reasoning with relationship graphs and symbolic logic remains limited. XDR vendors have marketed this capability more aggressively, positioning themselves as solving the "semantic gap" problem, but implementation depth varies significantly.

Option 3: As a Separate Analytics Layer

Building a semantic layer that operates alongside or on top of your security data infrastructure.

Advantages:

• Flexibility to model your specific environment

• Can work across multiple data sources

• Optimized for complex reasoning

• Natural fit for AI/ML integration

Challenges:

• Another platform to integrate and maintain

• Potential latency for real-time use cases

• Requires data federation or complex APIs

• Organizational boundaries (who owns it?)

Verdict: This is where most successful implementations end up, but it requires significant investment and organizational commitment. Some SOAR platforms have evolved toward this model, building semantic layers with automation workflows and embedding business context in case management. AI SOC platforms are also exploring this space, though many are still in early stages of true semantic reasoning versus simple enrichment.

Option 4: Distributed Semantic Processing

The most ambitious approach - semantic capabilities distributed throughout your stack, with processing happening wherever it makes most sense.

Advantages:

• Process data where it makes most sense

• No single point of failure

• Can optimize for different use cases

• Scales naturally with infrastructure

Challenges:

• Consistency across distributed models

• Complex orchestration and governance

• Difficult to maintain and update

• Requires sophisticated engineering

Verdict: This is the "microservices of semantic layers" - sounds great in theory, nightmarish in practice for most organizations. You need mature DevOps practices and significant engineering resources to make this work.

How Semantic Layer Projects Fail

Let's be honest about the failure modes, because understanding these is more valuable than any architecture diagram:

1. Trying to Model Everything

Organizations attempt to create comprehensive ontologies of their entire environment. This is impossibly complex and never finishes. You don't need a complete knowledge graph of your infrastructure to get value from semantic layers. But you do not explicit information about what is modeled and what is not.

2. Ignoring Organizational Reality

Semantic models require agreement on basic concepts like "what is a critical asset?" Most organizations can't even agree on this informally, much less formally model it. The technical challenge is often easier than the political one.

3. Underestimating Maintenance

Semantic models aren't write-once. They require constant updates as your environment evolves. Without dedicated resources, they become stale and useless within months.

4. Over-Engineering the Solution

Building elaborate graph databases and reasoning engines when simple lookup tables would suffice for 80% of use cases. Start with the minimum viable semantic layer that solves your specific problems.

5. Lacking Clear Use Cases

Implementing semantic layers because they sound advanced, not because they solve specific problems. "We need better context" isn't a use case - "We need to automatically identify which database access is anomalous based on team structure and data classification" is.

The Practical Path Forward

If you're considering implementing semantic capabilities, here's what actually works:

Start Small and Specific

Pick one use case with clear business value. User-to-asset relationship modeling for access analysis. Application dependency mapping for incident scope. Data classification for DLP prioritization. Prove value before expanding.

Embrace Imperfect Models

Your semantic layer doesn't need to be complete or perfect. Even modeling 60% of your critical relationships provides massive value over having none. Accept that edge cases will exist.

Build for Maintenance

Plan for model updates from day one. Who owns entity definitions? How do relationships get updated? What's the approval process for changes? The technical implementation is secondary to the governance model.

Leverage Standards Where Possible

Don't reinvent entity definitions that already exist. Frameworks like OCSF provide common semantic models that reduce custom development. Think of these as starting points, not constraints.

Consider Buy vs Build Carefully

Building custom semantic layers requires significant engineering investment. Many organizations would be better served by vendor solutions, even with their limitations. Be honest about your capabilities and resources.

As well some great resources:
https://www.holistics.io/books/setup-analytics/data-modeling-layer-and-concepts/

https://www.datacamp.com/blog/semantic-layer

https://arxiv.org/html/2506.17512v1

https://arxiv.org/html/2510.16610v1

Speculation on The Future of Semantic Layers in Security

Looking ahead, semantic layers will evolve in three key directions:

Standardization

Common semantic models will reduce the need for custom development. We'll share entity definitions and relationship structures like we share threat intelligence today. OCSF and similar frameworks are laying this groundwork.

Automation

ML will help maintain semantic models by learning from patterns and proposing updates. Instead of manually defining every relationship, you'll validate what the system discovers. Humans will shift from creation to curation.

Distributed Intelligence

Rather than centralizing all semantic reasoning in one layer, we'll see semantic capabilities embedded throughout security infrastructure. Your EDR will understand user context, your SIEM will reason about asset relationships, your SOAR will make decisions based on business impact. The semantic layer becomes infrastructure, not a product.

The organizations that succeed won't be those with the most sophisticated semantic models, but those that solve specific problems with appropriately scoped semantic reasoning. Start with Jerry and his VPN, not with a complete ontology of your enterprise.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

This is Part 2 of the SOC Reality Check series. In Part 1, I explained how we've made incredible progress on the middle (investigation/triage) and we're improving the left (detection engineering), but this success revealed a new bottleneck: the right side of the IR cycle. Now let's talk about what it actually takes to solve it, and what becomes possible when you do.

Let me start where Part 1 ended: We're actually winning at AI-powered security operations.

The middle is getting better and better, and the left is improving. But the right side, forensics, response, and recovery, is still the bottleneck.

In Part 1, I showed you what that bottleneck looks like. But here's what I didn't talk about: what becomes possible when you solve it.

Because the right side isn't just about faster response. It's about unlocking an entirely different way of operating your SOC, one where your team actually has time to hunt for threats proactively, validate your detections continuously, and improve your security posture measurably.

This edition is sponsored by Binalyze

Investigate cyber threats in minutes

AI-powered speed. Human-driven insight.

Binalyze AIR is the forensic investigation automation platform accelerating incident response with AI precision – fast. 

Learn more at - [Link]

Why AI Alone Can't Drive Response

Let me be very clear: AI is incredibly valuable for the right side, but only if you have the forensic automation infrastructure underneath it.

I keep seeing vendors pitch "AI-driven response" and "autonomous remediation." Here's what they usually mean: automated containment like blocking an IP, quarantining an endpoint, or disabling a user account.

Here's what they don't mean: Actually collecting forensic evidence. Understanding the full scope of the compromise. Hunting for related activity across the environment. Preserving artifacts before they're overwritten.

AI can help you decide what to investigate and recommend actions. It can correlate patterns and suggest attack paths. That's the brainwork, and AI is genuinely good at it.

But AI cannot collect memory dumps before the evidence is overwritten. It can't preserve forensic artifacts across Windows, Linux, macOS, and cloud consistently. It can't ensure chain of custody. It can't guarantee complete evidence instead of just whatever happened to be available.

AI is the brain that decides what to do. Forensic automation is the hands that actually do it. You need both, and right now most organizations only have the brain.

What Forensic Automation Actually Enables

Most people think forensic automation is just about "collecting evidence faster", cutting the collection time from 4 hours to 15 minutes.

That's only 20% of the value. The other 80% is what it unlocks downstream.

You Can Finally Afford to Be Thorough

Your AI SOC platform does an amazing job. It investigates 1,000 alerts per day, closes 700 as benign, and escalates 100 as True Positive or Inconclusive. Your dashboards look fantastic.

But here's what's actually happening: Your team looks at those 100 escalated alerts and realizes manual evidence collection takes 2-4 hours per incident. With 6-8 analysts on shift, they can properly investigate maybe 10-15 incidents with complete forensics.

So what happens to the other 85? You triage, you prioritize, you make judgment calls. "This one is probably just noise, close it." "This one is non-critical, deprioritize."

You're rationing investigation capacity.

Now, imagine the math changes completely. Evidence collection happens automatically in 5-15 minutes, triggered by alert severity. By the time an analyst looks at the incident, all forensic artifacts are already collected and waiting.

Suddenly you can be thorough with everything. You're no longer choosing which threats are "worth" deep investigation, you investigate all of them.

Related to this blog, there is a podcast episode, check it out:

You Can Validate Your AI's Accuracy

Here's what keeps me up at night: How do you actually know if your AI is making correct decisions?

Of those alerts your AI closed as "benign" with 95% confidence, how many did you validate with forensic evidence? Most organizations have no idea. The AI says "95% confident," you trust it (because you don't have capacity to investigate manually), and the alert gets closed.

This is flying blind with extra steps.

With automated forensic collection, you can close this loop. AI flags True Positive → Forensics collected automatically → Human validates with evidence → Results feed back to improve AI. Without forensic automation, this loop is broken.

From Reactive Firefighting to Proactive Hunting

When your team isn't drowning in manual evidence collection, something amazing becomes possible: they can finally do proactive work that actually improves security posture.

Let me show you what I mean.

The Reactive SOC: Where Most Teams Are Stuck

Your analyst arrives at 8:00 AM, checks overnight alerts. Spends the morning manually collecting evidence, SSH into systems, running scripts, hoping logs haven't rotated. By lunch, they've collected partial evidence from a few systems. Afternoon brings more alerts, more manual collection, more coordination. By 5:00 PM, they go home exhausted.

Time spent on proactive threat hunting: zero.

The Proactive SOC: What Becomes Possible

Same analyst, different infrastructure. By 9:00 AM, they've reviewed pre-collected evidence for all overnight incidents. Five are malicious, containment actions executed. Three are false positives, feedback logged. By 10:00 AM, incidents are handled.

Now they actually have time for proactive work.

10 AM to 12 PM: Hypothesis-driven threat hunting. 1 PM to 3 PM: Detection engineering based on findings. 3 PM to 5 PM: Purple team validation exercises.

5:00 PM: They go home having actually improved security posture, not just responded to what already happened.

Real-World Example: Hunting for Glassworm

Let me walk you through a real scenario. In January 2025, security researchers disclosed Glassworm, the first self-propagating worm using "invisible code" that hit VSCode marketplaces. This is a supply chain attack targeting developer environments.

What makes Glassworm nasty: it spreads through malicious VSCode extensions that look legitimate. Malicious code is hidden using zero-width characters. Once installed, it can steal credentials, propagate through git repositories, and establish persistence.

The problem: Most organizations have zero visibility into what VSCode extensions developers install. It's not something EDR monitors closely. Developers install extensions all the time, it's normal behavior.

The Reactive Approach: Waiting for Something Bad to Happen

Someone reads about Glassworm on Twitter. "We should probably check if we're vulnerable." Added to backlog.

Two weeks later, a developer tickets IT: "My system is acting weird." The ticket sits in the queue for days, it's not high priority.

Eventually, an analyst investigates. They check EDR logs for that one system. VSCode is making unusual network connections, but is that malicious? They don't have forensic artifacts to tell. They ask the developer what extensions are installed, the developer lists five they remember, but there are actually twelve installed.

How do you investigate a VSCode extension? They're just directories of JavaScript files. Is the code malicious? This investigation is going nowhere.

Meanwhile, the worm has been propagating for three weeks through git repositories. Other developers got infected. Nobody knows the full scope.

This is what happens when you're purely reactive.

The Proactive Approach: Hunting Before It Becomes an Incident

It's Monday morning, January 20th. The security team reads about Glassworm disclosed over the weekend. Instead of adding it to backlog: "Let's hunt for this right now."

Now, you might be thinking: "Wait, can't I do this hunt with my EDR?" And you'd be partially right. Yes, your EDR has some of this telemetry, process execution logs, network connections, maybe even some file system activity if you're collecting it.

But here's where it gets complicated:

Your EDR covers the 347 Windows and Mac developer workstations with agents installed. Great. But what about the 45 Linux developer boxes that use a different EDR agent with a different console? Or the 20 cloud development environments that don't have traditional endpoints? Or the containerized dev environments that spin up and down?

You're already in three different tools.

Now let's say you start hunting in your EDR console. You query for VSCode process execution across all endpoints. Your EDR retention is 90 days if you're lucky (and paying premium for that). The Glassworm disclosure mentions extensions installed three months ago. You might be outside your retention window for the initial infection.

You find some VSCode processes making unusual network connections. Good! But to investigate the actual extension code, you need file system artifacts, the extension directories, the JavaScript files, the package.json configs. Does your EDR collect that level of disk forensics? Maybe for specific directories you've configured, but did you tell it to monitor VSCode extension folders six months ago? Probably not.

Now you want to check persistence mechanisms. You pivot to your SIEM to search Windows event logs for scheduled tasks. Oh, but you need to correlate those with the specific VSCode processes you found in EDR. You're copy-pasting process IDs and timestamps between tools.

You want to see what git repositories the infected systems accessed, because the worm propagates through git. That data might be in your network monitoring tool or proxy logs. Now you're in a fourth tool, trying to correlate timestamps again.

And let's say you find evidence of compromise. Now you need to preserve it properly for potential legal proceedings with proper chain of custody. Your EDR wasn't really designed for that, it's a detection and response tool, not an evidence management system.

This is what I mean by "might be limited and you need to jump to multiple tools."

Why Unified Forensic Automation Changes This

With forensic automation platform collecting data across all your systems, not just endpoints with EDR agents, the hunt looks different:

First query: What VSCode extensions are installed across all developer systems?

One query. One platform. Covers Windows, Mac, Linux, cloud instances, containers, everything. Results in minutes: 1,240 unique extensions across 412 total systems (not just the 347 your EDR covers).

The forensic platform has been collecting file system artifacts continuously, not just process executions, but actual extension files, configuration, modification timestamps. You can see exactly what's installed, when it was installed, and examine the actual code if needed.

Second query: What network connections are VSCode processes making?

Same platform. Network forensics correlated automatically with process data. No pivoting to another tool. You're seeing unusual patterns on 12 systems.

Third query: What persistence mechanisms were created by VSCode processes?

Same platform. Scheduled tasks, startup items, cron jobs, all correlated with the processes that created them. You don't need to manually match timestamps across three different tools.

All of this in one investigation workspace. The forensic artifacts are preserved with proper chain of custody. Retention isn't limited to 90 days, you have historical data going back as far as you need for compliance.

You find 12 compromised systems across all your development infrastructure, including three Linux boxes and two cloud instances your EDR doesn't cover.

This is why "EDR has logs" isn't the complete answer. EDR is fantastic for what it does, but it's not designed to be a comprehensive forensic investigation and hunting platform. It's designed for endpoint detection and response.

The Tool Sprawl Problem

Here's what hunting looks like in most organizations today, even with good EDR:

• EDR console: Process execution, network connections (for systems with agents)

• SIEM: Log aggregation, scheduled tasks, authentication events

• Network monitoring tools: Proxy logs, DNS, NetFlow

• Identity systems: Account access patterns, privilege changes

• Cloud security tools: Cloud workload activity

• Vulnerability scanner: Patch status, software inventory

When you hunt, you're pivoting across all of these, manually correlating timestamps and indicators. Copy-paste investigation. It works, but it's slow, error-prone, and doesn't scale when you need to hunt across thousands of systems in two hours.

Forensic automation platforms consolidate this. Not by replacing your EDR or SIEM, but by providing a unified collection and investigation layer on top of them. You're still using your EDR for real-time detection and response. But when you need to hunt or do deep investigation, you have one place where all the forensic artifacts live, properly correlated, with extended retention.

The Detection Engineering Payoff

After the hunt, the team asks: "Why didn't our detections fire for this?" They had EDR monitoring, but weren't specifically looking at VSCode extension installations or obfuscated code execution from extension directories.

New detection built: Monitor for VSCode extensions from non-verified publishers making unusual network connections or creating persistence. Tuned using the forensic data they collected, tested against the twelve confirmed infections and 335 clean systems.

This is security posture improvement in action, each hunt makes your detection coverage broader and your blind spots smaller.

The Cultural Transformation

Solving the right side requires a fundamental shift in how SOC teams think about their work.

Most SOCs operate with a reactive mindset: Alerts come in, you respond. Success = how fast you clear the queue. You're constantly behind, always catching up.

When you solve the right side with forensic automation, you can shift to a proactive mindset: You assume threats are already in your environment that your detections missed. Your job is to actively hunt for them. Success = how much you improve detection coverage and validated security controls.

This shift doesn't happen automatically. It requires leadership commitment, restructuring how analyst time is allocated (dedicating 20-30% to hunting), changing what metrics you track, and training your team on hunting methodologies.

But the organizations that make this shift? They're the ones who find supply chain attacks like Glassworm before they spread through the entire organization.

The Practical Implementation Path

Phase 1: Build the Forensic Automation Foundation

Everything depends on this. You need a system that can remotely collect comprehensive forensic evidence across all OS types, triggered automatically based on alert severity, preserving chain of custody properly.

This doesn't mean ripping out your EDR. Your EDR is still critical for real-time detection and response. But you need a forensic layer that:

• Collects deeper artifacts than typical EDR telemetry (disk forensics, memory dumps, full file system analysis)

• Covers systems beyond traditional endpoints (cloud instances, containers, systems without agents)

• Provides extended retention without the cost of expanding EDR storage

• Unifies investigation across all your security tools in one workspace

• Preserves evidence properly for legal and compliance needs

Tools like Binalyze are purpose-built for this problem. You're not replacing your SIEM or EDR, you're filling the gap between "alert fired" and "comprehensive evidence collected for hunting and investigation."

Phase 2: Free Up Analyst Time for Hunting

Allocate dedicated hunting time (20-30% of each analyst's week). Make it protected time, not "hunt when you have spare cycles." Define a hunting cadence, weekly TTP-based hunts, monthly purple team exercises, quarterly baseline deviation hunts.

Phase 3: Close the Feedback Loops

From hunting to detection engineering: What gaps exist? What new rules should be built? The Glassworm hunt should result in new detections for supply chain attacks.

From response to AI improvement: Was the AI's verdict correct? Validate with forensic evidence.

From forensics to threat intelligence: Share what you're actually seeing in the wild.

The Bottom Line: Beyond "Faster Response"

The right side isn't just about responding faster to incidents. It's about fundamentally transforming how your SOC operates.

When you solve the right side with forensic automation, the response becomes faster and the evidence is complete. But those are just the table stakes.

The real transformation is that proactive hunting becomes feasible. Your team can actually hunt for threats when intel drops, instead of hoping your detections catch it. Detection engineering gets continuous feedback. Your security posture improves measurably over time.

Yes, your EDR gives you telemetry. Yes, you can do some hunting with it. But when you need to hunt across your entire infrastructure, endpoints, cloud, containers, systems without agents, and correlate findings without pivoting through five tools, that's where forensic automation makes the difference.

This is the shift from a SOC that fights fires well to a security operation that prevents fires from starting. From reactive response to proactive defense. From hoping your detections work to validating they work and continuously improving them.

We did great work solving the middle with AI triage. We're making real progress on the left with better detection engineering. But the right side is what unlocks the SOC everyone actually wants to work in, one that hunts threats proactively and improves security posture measurably.

Don't just close alerts faster. Build a SOC that actually gets ahead of threats.

That's what the right side is really about.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

A two-part series on the new bottleneck emerging in AI-powered SOCs, and why solving investigation created a better problem to solve.

Let me start with some good news: We're actually winning at AI-powered security operations.

No, really. Hear me out.

Over the last two years, AI has genuinely transformed the middle of the IR cycle, investigation and triage. Vendors build amazing AI SOC analysts that can handle the 5Ws (Who, What, When, Where, Why), enrich alerts with context, and reach accurate verdicts in seconds instead of hours.

The middle is getting solved. AI can investigate alerts at scale, and it does it well (of course if you implement it right)

And now? The industry is shifting left. We're seeing platforms tackle data pipelines, log normalization, and even detection engineering itself. AI is helping us build better detections, optimize coverage, and reduce false positives at the source.

This is real progress. We should celebrate it.

But here's what's happening as a result of this progress: we're moving the bottleneck.

This edition is sponsored by Binalyze

Investigate cyber threats in minutes

AI-powered speed. Human-driven insight.

Binalyze AIR is the forensic investigation automation platform accelerating incident response with AI precision – fast. 

Learn more at - [Link]

The Problem We Created by Solving the Middle

When AI handles investigation and triage brilliantly, and when detection engineering improves to generate higher-fidelity alerts, you end up with a new challenge:

You now have more True Positives and Inconclusive alerts that require deep investigation and incident response.

Let me break this down with a real example:

Before AI (the old bottleneck):

• 1,000 alerts per day

• Analysts can investigate maybe 50-100 per day

• 900 alerts never get properly looked at

• Real threats hiding in the noise

• Bottleneck: Can't investigate everything

After AI in the Middle (current state):

• 1,000 alerts per day

• AI investigates all 1,000

• AI closes 700 as benign false positives

• AI flags 200 as low-confidence, likely benign

• AI escalates 100 as True Positive or Inconclusive

• Analysts now have 100 high-quality alerts that need deeper investigation

• New bottleneck: Can't properly respond to everything

See what happened? You went from "drowning in alerts" to "drowning in incidents that need forensic investigation and response."

This is actually a better problem to have, you're working on real threats now, not noise. But it's still a bottleneck.

Related to this blog, there isa podcast episode, check it out:

But Here's Where It Gets Interesting

While the bottleneck has clearly shifted right, smart organizations are discovering that forensic automation platforms can actually enhance investigation at every stage, not just the deep forensic response. 

Even during AI triage, when your AI SOC analyst is determining whether that alert is a True Positive or False Positive, having instant access to forensic artifacts can dramatically improve verdict accuracy:

• Memory analysis reveals the full process tree AI couldn't see from logs alone

• Disk artifacts show persistence mechanisms that log data missed

• Network connection history provides context that makes "Inconclusive" → "True Positive"

This means fewer alerts stuck in "Inconclusive" limbo and more confident verdicts earlier in the process. The forensic infrastructure you build for deep investigation also makes your AI triage more effective.

What "Deep Investigation and Response" Actually Means

When your AI SOC analyst escalates a True Positive or marks something as Inconclusive, here's what actually needs to happen:

The AI hands you a verdict:

TRUE POSITIVE: Lateral movement detected

User: john.doe

Source: WORKSTATION-047  

Target: DC-01 (admin share accessed)

Confidence: HIGH

MITRE: T1021, T1570

Recommendation: Immediate forensic investigation and containment

Great! Now what?

Your analyst (or IR team) needs to:

1. Collect forensic evidence from affected systems
   

   • Memory dumps before artifacts are overwritten

   • Disk analysis for persistence mechanisms

   • Process execution history

   • Network connections and data transfers

2. Determine blast radius

   • What other systems did this user access?

   • Are there signs of lateral movement elsewhere?

   • What data might have been accessed or exfiltrated?

3. Preserve evidence for potential legal/compliance needs

   • Chain of custody documentation

   • Timeline reconstruction

   • All artifacts properly stored and indexed

4. Execute containment according to your IR playbook

   • Isolate affected endpoints

   • Disable compromised accounts

   • Block malicious IPs/domains

   • (All with proper approvals and governance)

5. Coordinate with stakeholders

   • IT for system access and changes

   • Legal for compliance and evidence handling

   • Management for business impact decisions

   • External parties if breach notification required

And here's the kicker: most of this is still manual, slow, and inconsistent.

Why the Right Side Became the New Bottleneck

As you solve the middle (investigation/triage) and improve the left (detection engineering), you're naturally generating more high-quality incidents that require the right side of the IR cycle to work properly:

Investigation → Containment → Eradication → Recovery → Lessons Learned

But the right side hasn't kept pace with the middle and left. Here's what's still broken:

1. Forensic Evidence Collection is Manual and Slow

What actually happens:

• Analyst gets Alerts that requires further investigation at 10:00 AM

• Starts trying to collect forensic evidence at 10:15 AM

• Realizes they need to SSH into multiple systems(or use your EDR)

• Discovers some logs have already rotated out (oops)

• Manually pulls what's available from EDR (if the agent is installed)

• Tries to get memory dumps (complicated, requires specific tools and permissions)

• By 2:00 PM, maybe has incomplete forensic evidence

The problem:

• Evidence gets overwritten or rotated before you can collect it

• Collection process is different for every analyst ("tribal knowledge")

• Manual processes don't scale when you have 20 incidents per day instead of 5

2. IR Playbooks Aren't Executable

Remember my blog series on playbooks? Most IR playbooks are PDFs or wiki pages that say things like:

Lateral Movement Response:

1. Isolate affected systems

2. Collect forensic evidence

3. Determine scope of compromise

4. Reset credentials

5. Document findings

This is guidance for humans, not executable automation.

When your AI SOC solution escalates 20 True Positives per day, you can't manually execute these steps 20 times. You need:

• Automated forensic collection triggered by alert severity

• Standardized evidence preservation

• Orchestrated response actions with proper governance

• Consistent execution regardless of who's on shift

3. Response Actions Require Too Much Coordination

Even when you know what needs to be done, executing it requires:

• Multiple approval chains (IT, Security, Management)

• Coordination across teams (Security, IT, DevOps)

• Navigating change management processes

• Fear of business disruption from containment actions

So containment that should happen in minutes takes hours or days because you're stuck in Slack threads and email chains getting approvals.

The Shift Map: Where We Are and Where We're Going

Let me map this against the SecOps AI Shift Map I introduced in my previous blog:

Left Side (Detection & Data):

• ✅ AI is starting to help here

• ✅ Better detection engineering with AI assistance

• ✅ Improved data pipelines and normalization

• Status: Progress being made

Middle (Investigation & Triage):

• ✅ AI SOC analysts handle this brilliantly

• ✅ Enrichment, 5Ws, verdict reached in seconds

• ✅ False positives filtered out effectively

• Status: Progress being made

Right Side (Response & Recovery):

• ❌ Forensic evidence collection still manual

• ❌ IR playbooks not machine-executable

• ❌ Response actions require too much human coordination

• ❌ Recovery and lessons learned feedback loops broken

• Status: This is the new bottleneck

Why This Matters: The Investigation Debt Problem

Here's a concept I don't hear talked about enough: Investigation Debt.

Investigation Debt is what accumulates when you have True Positives or Inconclusive alerts that you can't properly investigate with complete forensic evidence.

Every time you:

• Close a True Positive without collecting full forensic artifacts

• Skip deeper analysis because evidence collection is too manual

• Move on because "we don't have time to do full forensics on everything"

• Accept "the logs rotated out" as an answer

You're accumulating Investigation Debt.

And here's what makes this dangerous: that lateral movement alert you couldn't fully investigate three months ago might have been your initial compromise. But you didn't collect the forensic evidence, logs rotated, the attacker cleaned up, and now you're doing incident response in the dark.

The irony is that better AI in the middle makes Investigation Debt more visible.

Before AI, you had so much noise you didn't even know which alerts were real. Now AI is flagging possible True Positives for you, and you're realizing: "We don't have the capacity to properly respond to all of these."

What "Solving the Right Side" Actually Requires

So what does the right side need to keep pace with the middle and left?

1. Automated Forensic Evidence Collection

When a True Positive alert fires, evidence collection should happen automatically, not manually:

• Memory dumps captured before artifacts are overwritten

• Disk forensics collected immediately

• Process trees and network connections preserved

• All artifacts time-stamped and stored with proper chain of custody

This needs to be triggered by alert severity, not analyst memory.

2. Response Orchestration with Governance

Response actions need to be automated but with proper safety rails:

• Approval workflows for high-impact actions

• Blast radius checks (don't isolate 500 endpoints because one script failed)

• Rollback procedures if containment causes problems

• Audit trails of who approved what and when

This is where SOAR was supposed to help, but as I wrote in my Shift Map blog: we failed at SOAR not because the tech was bad, but because our processes were a mess.

The Path Forward: Shift Right

Here's where we are:

✅ Middle is getting solved - AI handles investigation and triage brilliantly

🔄 Left is improving - Detection engineering and data pipelines are getting AI assistance

❌ Right is the bottleneck - Forensic collection, response execution, and recovery are still manual

The solution isn't to slow down the middle or left. The solution is to build the right side infrastructure to match.

That means:

• Forensic automation platforms that collect evidence at machine speed

• Machine-executable IR playbooks that respond consistently

• Response orchestration with appropriate governance and safety rails

• Feedback loops that actually work (lessons learned → detection engineering)

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

A Quick Note Before We Start

When I wrote about how AI transforms detection engineering from narrow precision to broad coverage, I originally planned this as Part 2 of that series. The logic made sense: Part 1 shows how AI enables comprehensive detections, Part 2 would explain how to handle the resulting alert volume.

But as I started writing, I realized this piece actually belongs with a different blog I published a couple months ago: "Why SOC Analysts Ignore Your Playbooks".

Here's why: The detection engineering blog is about what you can now detect. This blog is about how you investigate what you detect. And the investigation problem traces back to the same root causes I identified in the playbooks blog: broken processes, ignored documentation, and tribal knowledge that walks out the door when your best analysts leave.

So consider this Part 2 of the playbooks series, not the detection engineering one. Though honestly, they're all connected; you can't deploy comprehensive detections without machine-executable investigation procedures, and you can't build those procedures if your playbooks are PDF documents nobody follows.

If you haven't read Part 1 yet, start there. It explains why playbooks are broken and how to fix them by coupling them with detections. This post takes that foundation and shows you how to transform those playbooks into logic that AI can actually execute.

Let's dive in.

The Problem We Thought We Solved

In Part 1, I argued that the biggest mistake in SecOps is shipping detections without playbooks. That shadow detection sitting in your SIEM that nobody understands? That's the root cause of your broken feedback loop, your alert fatigue, and your analyst burnout.

The fix seemed straightforward:

1. Build the playbook when you build the detection

2. Use GenAI to draft it in seconds

3. Automate the steps you can

4. Let AI learn from analyst behavior for the rest

And for human analysts, this works. You now have documented procedures, analysts can follow them (or at least reference them when needed), and new joiners can learn how your organization operates.

But then you try to hand these playbooks to an AI agent to execute autonomously.

And it all falls apart.

This edition is sponsored by Prophet Security

Put every alert through a complete investigation.

Prophet AI pulls the right context, follows a reproducible line of questioning, and returns a clear determination with linked evidence and an audit trail. Analysts stay in control while investigations finish faster and hold up under review. 

See it on your data. Request a demo at prophetsecurity.ai

Why Your "Fixed" Playbooks Still Break AI

Let's take a typical playbook that follows my advice from Part 1. It's not a dusty PDF,it's embedded with the detection, it's maintained, and it actually reflects how your team works:

Suspicious Login Investigation Procedure

When you receive a suspicious login alert, follow these steps:

1. Check whether the login location matches the user's typical locations

2. Review their login history for the past 30 days in your identity provider

3. If the location is unusual, verify whether the user recently traveled

4. Check if the device fingerprint is recognized

5. If multiple red flags exist, contact the user's manager

6. If you cannot confirm legitimacy within 2 hours, force password reset

This is a GOOD playbook by Part 1 standards. It's clear, actionable, tied to a specific detection. An experienced analyst knows exactly what to do.

But hand this to an AI agent, and watch what happens:

What exactly is an "unusual location"?

• 50km away? 500km? Different country? Different continent?

• What if the user works remotely and travels frequently?

• What if they're using a VPN that makes location data unreliable?

What constitutes "multiple red flags"?

• Two flags? Three? Which combinations matter more?

• Is unusual location + recognized device better or worse than normal location + unrecognized device?

• How do you weight these factors?

What if your identity provider is temporarily unavailable?

• Does the AI wait? For how long?

• Does it proceed without that data? How does that affect its confidence?

• Does it automatically escalate?

When should AI act autonomously vs. escalate to a human?

• Can it force a password reset on its own? For which user types?

• What if the "suspicious" login is the CEO accessing email at 2 AM?

Human analysts handle this ambiguity through experience, context, and judgment. They know that "unusual" for the CEO is different than "unusual" for a contractor. They know which data sources to trust. They adapt when systems are down.

AI doesn't have that context unless you give it explicitly.

The Missing Layer: Machine-Executable Logic

This is what I've realized since Part 1: there's a gap between what humans need and what AI needs.

Humans need:

• Clear guidance on what to investigate

• Reference points for common scenarios

• Flexibility to adapt based on context

• Your Part 1 playbooks solve this

AI needs:

• Explicit decision logic with no ambiguity

• Quantifiable thresholds and confidence scores

• Complete coverage of edge cases and fallbacks

• Structured entity resolution across data sources

• Your Part 1 playbooks don't solve this

So we need a new layer. Not to replace human-readable playbooks, but to sit underneath them.

Let me show you what this looks like.

The Same Playbook, But Machine-Executable

Here's how that suspicious login investigation transforms when designed for AI execution:

yaml
playbook: suspicious_login_investigation
version: 2.3
owner: identity_security_team

# Data source requirements with structured fallbacks
required_data_sources:
  okta_logs:
    retention_days: 30
    required_fields: [user_id, src_ip, location, device_fingerprint, mfa_status]
    unavailable_action:
      type: cap_max_confidence
      max_confidence: 0.80
  
  vpn_logs:
    retention_days: 7
    unavailable_action:
      type: confidence_penalty
      delta: 0.10

# Entity resolution: how to connect data across systems
entity_resolution:
  user_identity:
    canonical_identifier: email
  
  crowdstrike_endpoint:
    steps:
      - {action: map, from: alert.email, to: active_directory.userPrincipalName}
      - {action: get, field: active_directory.lastLogonComputer}
      - {action: map, from: active_directory.lastLogonComputer, to: crowdstrike.hostname}
      - {action: get, field: crowdstrike.AID}
    fallback: {strategy: use_cache, max_age_hours: 24}

# Confidence scoring: weights sum to 1.0
confidence_scoring:
  method: weighted_sum
  normalize_weights: true
  
  factors:
    - name: location_deviation
      weight: 0.30
      scoring:
        - {when: "distance_km < 50 AND time_plausible", score: 0.80}
        - {when: "distance_km >= 500 OR impossible_travel", score: 0.20}
    
    - name: mfa_status
      weight: 0.25
    
    - name: device_recognition
      weight: 0.25
    
    - name: time_of_day_anomaly
      weight: 0.15
    
    - name: concurrent_activity
      weight: 0.05

# Complete decision ranges (no gaps: 0.0-1.0 covered)
decision_thresholds:
  auto_close_benign: {range: [0.95, 1.00], action: close_with_docs}
  auto_close_fp: {range: [0.90, 0.95], action: close_and_tune_detection}
  low_risk_action: {range: [0.75, 0.90], action: require_mfa_reauth}
  escalate_medium: {range: [0.60, 0.75], action: analyst_review, sla_hours: 4}
  escalate_high: {range: [0.40, 0.60], action: analyst_review, sla_hours: 2}
  escalate_critical: {range: [0.00, 0.40], action: incident_response, sla_minutes: 30}

# Safety rails for autonomous actions
governance:
  suspend_account:
    approval_required: true
    two_person_rule: true
    blast_radius_check: true
  
  isolate_endpoint:
    approval_required: true
    blast_radius_limit: 5

# Audit with privacy controls
audit_trail:
  capture: [data_sources_queried, confidence_scores, reasoning_chain, action_taken]
  privacy: {redact_pii: true, respect_residency: true}
```

See the difference?

No ambiguity. "Unusual location" becomes distance_km >= 500 OR impossible_travel with a confidence score of 0.20.

No gaps. Every possible confidence score from 0.0 to 1.0 maps to a specific action.

Explicit fallbacks. If Okta is down, max confidence caps at 0.80. If VPN logs are unavailable, confidence drops by 0.10.

Measurable weights. Location deviation is weighted 0.30, MFA status 0.25, device recognition 0.25,you can validate these against historical outcomes.

Clear governance. AI can suggest account suspension, but requires human approval with two-person rule.

This isn't documentation for humans to read. It's executable logic for AI to run.

Connecting Back to Part 1: The Full Picture

Remember in Part 1 when I talked about two approaches to fixing playbooks?

• Approach 1: Build playbooks during detection engineering

• Approach 2: Let AI observe analysts and generate playbooks from behavior

Both of these still apply. But now I'm adding a critical step:

Approach 1 Extended: Detection Engineering → Machine Logic

1. Build human-readable playbook with detection (Part 1)

2. Identify which steps can be automated

3. Transform those steps into machine-executable logic (Part 2)

Approach 2 Extended: Behavioral Learning → Quantified Rules

1. Let AI observe how analysts investigate alerts (Part 1)

2. Generate initial playbook from behavior patterns

3. Codify the decision logic into structured, quantifiable rules (Part 2)

The insight from Part 1 still holds: analysts going by "instinct" have built better processes in their heads.

The challenge in Part 2 is: how do we capture that instinct as explicit, measurable logic that AI can execute?

This is where those analysts who ignore playbooks actually become your most valuable asset. Watch how they handle edge cases. Ask them why they made each decision. What data points did they weigh most heavily? When did they escalate vs. auto-close?

That tribal knowledge you were trying to capture in Part 1? Now you're quantifying it for Part 2.

What This Enables (Beyond Just Automation)

When I published Part 1, the focus was on fixing broken processes. If playbooks are ignored, your feedback loop breaks, your detection tuning stops, and analyst burnout accelerates.

But machine-executable playbooks unlock something bigger: measurable, improvable AI decision-making.

In Part 1, I mentioned the vanity metrics problem, everyone obsesses over alert closure times, but nobody tracks whether AI is making correct decisions.

Machine-executable playbooks make real metrics possible:

AI Decision Accuracy

• Of benign closures, how many were actually benign?

• When AI says 95% confident, is it right 95% of the time?

• Which alert types show the most false positives from AI triage?

Learning Over Time

• Does accuracy improve week over week?

• Do analyst overrides result in playbook improvements?

• Is the system getting better at handling your specific environment?

Drift Detection

• Is AI accuracy degrading as your environment changes?

• Which confidence factors show the most drift?

• Are updates keeping pace with infrastructure changes?

You can't measure any of this with ambiguous playbooks that analysts interpret differently on every shift. But with explicit, structured logic? You can track every decision, validate every confidence score, and continuously improve.

This is the feedback loop Part 1 was trying to fix, now operating at AI speed.

The Six Requirements for Machine-Executable Playbooks

Based on everything I've learned building these for real environments, here's what actually matters:

1. Explicit Decision Logic (No Ambiguity, No Gaps)

Your decision thresholds must cover every possible confidence score from 0.0 to 1.0. No "if unusual, escalate" nonsense. Define EXACTLY what triggers each action.

❌ Bad: If suspicious, escalate to analyst

✅ Good:

2. Data Topology and Entity Resolution

Your AI needs to know how to connect data across your specific tool stack. How do you go from alert email → Active Directory → CrowdStrike endpoint ID?

This was implicit tribal knowledge in Part 1's world. In Part 2, it must be explicit:

3. Confidence Calibration (Validated Against Outcomes)

Remember in Part 1 when I said the feedback loop is broken because analysts ignore the step that says "send false positives back to detection engineering"?

Machine-executable playbooks fix this by requiring outcome validation:

• If AI says 90% confident benign, track how often it's actually benign

• Adjust confidence weights based on historical accuracy

• Recalibrate when you detect drift

The confidence scores must be trustworthy, not aspirational.

4. Reproducible Metrics and Evaluation

When a vendor claims "improves coverage from 50% to 95%," ask: "95% of what, measured how?"

With machine-executable playbooks, you can test against:

• Historical incidents with known outcomes

• Purple team exercises covering MITRE ATT&CK techniques

• Adversary emulation with documented expected results

Define variant coverage as: (# of distinct attack technique variants correctly triaged) / (total curated variants in test set)

This separates real capability from marketing claims.

5. Safety Rails and Governance

In Part 1, I talked about how deterministic automation needs approval workflows. The same applies here, but even more critically.

AI Agents handle: Investigation, enrichment, triage decisions, confidence scoring, and recommending next steps

Deterministic automation handles: Response execution with defined approval workflows, rate limiting, blast radius checks

Why this matters: AI reasoning excels at weighing ambiguous evidence during investigations. However, response actions need predictability, auditability, and fail-safe controls.

Governance controls:

• Confidence thresholds determine when AI can auto-close vs. escalate

• Escalation criteria define what triggers human involvement

• Override tracking captures when analysts disagree with AI (this feeds back into learning)

6. Complete Observability and Transparency

In Part 1, I said the problem with playbooks is they're ignored and never maintained. Part 2 solves this by making every AI decision traceable:

• Complete audit trails showing data queried, confidence calculated, thresholds applied

• Reasoning chains explaining every decision step

• Metrics tracking accuracy, drift, and performance over time

• Testing frameworks for validating playbook changes before deployment

Black boxes are unacceptable. If you can't see how AI reaches conclusions, you cannot trust it, improve it, or measure its effectiveness.

This is how you prevent Part 2 from becoming Part 1 all over again,broken processes that nobody maintains.

Two Paths to Implementation (Revisited from Part 1)

In Part 1, I outlined how to build playbooks alongside detections. Now let's extend that with two approaches to machine-executable logic:

Route 1: Customizable Intelligence

The approach: Build your own machine-executable playbooks for your specific environment. You control procedures, entity mappings, confidence weights, and thresholds.

Best for: Mature security operations with established procedures, engineering resources to invest, and unique tool stacks requiring customization.

What you get: Maximum control to encode your specific tribal knowledge and operational context,the "instinct" your best analysts use.

What it requires: Security engineering resources to build, test, and maintain playbooks over time. This is the formalization of the behavioral learning I mentioned in Part 1.

Route 2: Out-of-the-Box Intelligence

The approach: Pre-trained AI with built-in investigation procedures based on security best practices and collective intelligence across many deployments.

Best for: Faster time-to-value, teams lacking resources to build custom playbooks, organizations wanting proven procedures.

What you get: Working baseline from day one, proven procedures, faster deployment.

What you can customize: Thresholds, escalation criteria, and specific organizational context as you learn what works.

The Non-Negotiable Requirement

Regardless of path, transparency is mandatory.

Remember the RPA problem I mentioned in Part 1? RPA solutions failed in cybersecurity because the tech wasn't there and the environment was too unpredictable.

AI solves the unpredictability through non-deterministic reasoning. But only if you can:

✓ See the investigation logic and understand what the AI does
✓ Trace the complete reasoning chain from alert to decision
✓ Audit all decisions with confidence scores and data sources
✓ Modify procedures as your requirements change
✓ Measure effectiveness with accuracy, learning, and drift metrics

Critical questions to ask any AI SOC platform:

• Can I see the investigation logic being applied?

• How are confidence scores calculated?

• What happens when data sources are unavailable?

• How do you handle entity resolution across my specific tools?

• Can I test changes before deploying them?

• How do you measure and track accuracy over time?

Without answers to these, you're just automating chaos faster.

The Bottom Line: Completing the Transformation

In Part 1, I showed you why playbooks are broken and how to fix them by coupling them with detections. That solves the human problem.

But Part 2 is the necessary evolution: those human-readable playbooks don't work for AI execution.

Machine-executable SOPs bridge the gap between broader detections and manageable operations. They're what allow AI to triage thousands of alerts with the same quality your best analyst applies to dozens.

This isn't better documentation. It's a fundamental shift in how security knowledge gets operationalized,moving from ambiguous human guidelines to explicit, testable, measurable logic that can be continuously improved.

The connection to Part 1:

• Analysts ignore playbooks → Build them with detections

• Detections without playbooks break feedback loops → Automate what you can

• Analysts go by instinct → Capture that tribal knowledge

• Now codify that knowledge into logic AI can execute → Measure and improve systematically

The six requirements that matter:

1. Explicit decision logic with complete coverage (no gaps, no ambiguity)

2. Data topology and entity resolution (connecting data across your specific systems)

3. Confidence calibration (trustworthy thresholds validated against outcomes)

4. Reproducible metrics (defensible claims about coverage and accuracy)

5. Safety rails (governance preventing business disruption)

6. Complete observability (transparency enabling trust and improvement)

Whether you build custom playbooks or start with proven out-of-the-box intelligence, these requirements don't change. The platform must provide them, and you must be able to validate they're working in your specific environment.

Without Part 1, your processes stay broken.

Without Part 2, you're automating chaos faster.

With both, you're building a security operation that measurably improves over time.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Part 1 of 3: The Detection Coverage Problem and How AI Solves It

Your SOC processes ten thousand alerts daily. Your detection engineer just wrote a brilliant new rule detecting lateral movement via WMI, but here’s what happens next:

They look at the alert volume and realize it generates two hundred potential hits per day. They know your team can realistically investigate maybe twenty alerts per day for this detection type, so they start making the rule more restrictive. They add filters, raise thresholds, and narrow the scope until the alert volume drops to something manageable.

In doing so, they’ve just created a blind spot.

Those one hundred and eighty alerts they filtered out might contain real threats, but your process design forced them to choose between overwhelming the team and potentially missing attacks.

This is the fundamental problem we need to solve. Your processes were designed for human-in-the-loop execution, and that constraint is now the bottleneck strangling your security effectiveness.

This edition is sponsored by AiStrike

AI SOC Done Right!

AI SOC Intelligence Fabric that unifies your data, accelerates investigations, and orchestrates intelligent response

Transform your SOC with composite AI that arrives pre-trained and ready to work. Enable small teams to operate like enterprise SOCs while giving enterprises state-of-the-art incident response capabilities.

What changed today

Today's AI wave is not a plug-and-play upgrade for security operations. Just as the shift to cloud and SaaS forced organizations to realign processes, roles, and governance, the AI wave demands a full reboot of your people-process-technology stack.

This isn't about adding a new tool to your existing workflows. This is about fundamentally rethinking how security operations function when you remove the human throughput constraint from the equation.

The Research Backs This Up

Recent research reveals an uncomfortable truth: data quality predicts success more than raw technology capacity, and process design often outweighs management intent in driving integration [1]. Meanwhile, cybersecurity researchers are exploring human-AI co-teaming models in SOCs, stressing the need for dynamic autonomy, trust calibration, and feedback loops in operational workflows [2].

The crux: Dropping AI into a rigid SOC is like installing a jet engine on a cart with square wheels. The power is there, but the system isn't designed to harness it.

What's Really Changing

Every major technology wave forces security teams to renegotiate the relationship between people, process, and technology:

• Cloud era: Reinvented access models, monitoring pipelines, identity governance

• SaaS era: Adapted to distributed ownership and ephemeral infrastructure

• AI era: Must handle systems that don't just observe, they act, decide, and recommend

The challenge isn't visibility anymore. It's an agency. AI systems don't just help us monitor threats; they investigate, triage, and recommend actions. That shift means SOC processes can't remain static, checklists written for human cognition. They must become machine-executable logic that adapts to model confidence, context, and risk.

The Brutal Trade-Off: Killing Your Detection Coverage

Traditional detection engineering operated under constraints that forced you to sacrifice coverage for operational feasibility. Let me show you what this looks like in practice.

The Typical Detection Engineering Process

Here's how it actually works:

1. Developp hypothesis about a threat you want to detect

2. Build a detection rule to identify that behavior

3. Test against your environment to see alert volume

4. See 100 alerts per day 😰

5. Realize your team can only handle 20 alerts per day

6. Make the detection more restrictive (add filters, raise thresholds)

7. Deploy detection that catches 40% of attack variants instead of 90%

You weren't optimizing for security effectiveness. You were optimizing for operational survival.

Understanding the Funnel of Fidelity

Zack Allen in the Detection Field Manual #3 talks about detection efficency concept of the Funnel of Fidelity (introduce by Jared Atkinson back in 2019)  to describe this exact problem[3]: massive data volume at the top, limited analyst capacity at the bottom. Every alert that survives the funnel consumes human focus, creating an inherent trade-off between comprehensive detection and operational sustainability.

This creates a dangerous dynamic. You might achieve eighty percent detection coverage, meaning your rules can theoretically identify eighty percent of relevant security events in your environment. However, analyst capacity constraints mean you can only thoroughly investigate fifty or sixty percent of the alerts those detections generate.

Your effective security coverage isn't 80%, it's the 40-50% that actually receives quality investigation.

The Attacker's Advantage

This coverage gap becomes an exploitable vulnerability. Attackers need only operate in the 20-30% of alert volume that your team doesn't have the capacity to investigate. They can:

• Generate low-level alerts that get suppressed automatically

• Operate during high-volume periods when your team is overwhelmed

• Use techniques that generate alerts your team habitually ignores due to high false positive rates

The gap between what you detect and what you investigate is where attackers live.

How AI Changes the Detection Engineering Equation

When investigation capacity increases from 20 alerts per analyst per day to thousands of alerts per AI agent per day, everything changes. You can finally deploy the detections you always wanted to build.

Before vs. After: The Transformation

The Key Shift

With AI, you move from precision-optimized detection to coverage-optimized detection.

Precision-optimized (old way):

• Question: "How can I make this detection narrow enough to be sustainable?"

• Result: Restrictive filters, high thresholds, missed attack variants

• Coverage: 30-40% of the actual threat landscape

Coverage-optimized (new way):

• Question: "How can I make this detection broad enough to catch all variants while maintaining signal quality?"

• Result: Comprehensive coverage, AI handles triage burden

• Coverage: 85-95% of the actual threat landscape

The detection engineer's job transforms completely. Instead of adding restrictive filters to reduce volume, she focuses on adding context that helps the AI make accurate disposition decisions. Instead of tuning for low volume, she tunes for high recall, knowing the AI can handle the resulting triage burden.[4]

Understanding the Alert Categories

(Updated section : Thank you Nathan Eades for the feedback)
When we talk about the triage burden, we're actually dealing with two distinct categories:

Benign Alerts: The detection is working correctly; it identified the behavior it was designed to catch. But the activity is legitimate, authorized, or expected.

• Example: Your lateral movement detection correctly flags WMI activity, but it's authorized IT maintenance during a change window

• Problem: Requires context to distinguish legitimate from malicious

False Positives: The detection is firing incorrectly due to overly broad rules or environmental noise.

• Example: Your detection fires on normal admin behavior because it doesn't account for privileged user patterns

• Problem: The detection rule itself needs tuning

Traditional SOCs struggled with both:

• Benign alerts required manual context gathering (check change tickets, verify with user, confirm authorization)

• False positives required detection tuning, but that tuning often meant narrowing the rule and missing real threats

AI handles both categories intelligently:

• For benign alerts: AI gathers context automatically (change windows, user roles, business justification) to determine legitimacy

• For false positives: AI identifies systematic patterns and suggests detection improvements

The result: You can deploy broader detections because AI can distinguish between malicious activity, benign activity, and false positives at scale.

The Transformation in Detection Philosophy

This isn't just about automation making things faster. It's a fundamental shift in how you approach detection engineering.

Traditional Detection Engineering

Guiding questions:

• Will this detection generate too many alerts?

• Can our team handle the volume?

• How can I make this more restrictive without losing too much coverage?

Optimization goal: Operational sustainability

Trade-off: Coverage sacrificed for precision

Result: Narrow detections that miss attack variants

AI-Enabled Detection Engineering

Guiding questions:

• Does this detection catch the full breadth of attacker behavior?

• What context does AI need to make accurate triage decisions?

• How can I optimize for recall without sacrificing signal quality?

Optimization goal: Security effectiveness

Trade-off: Let AI handle triage burden, focus on coverage

Result: Broad detections that catch attack variants while maintaining a manageable analyst workload

Key Metrics That Change

Traditional metrics focused on volume management:

• ✓ Alerts per day per detection

• ✓ Analyst handling capacity

• ✓ Alert-to-incident ratio

New metrics focus on coverage and learning:

• ✓ Detection recall: Of all malicious events, how many did we catch?

• ✓ AI triage accuracy: Of AI's auto-close decisions, what percentage are correct?

• ✓ Analyst amplification: How many alerts can each analyst effectively handle with AI assistance?

• ✓ Feedback utilization: Is analyst feedback improving AI accuracy over time?

What This Means for Your SOC

The transformation from narrow, precision-focused detections to broad, coverage-optimized detections has implications that ripple through your entire security operations:

For Detection Engineers

New responsibilities:

• Build comprehensive detections without volume anxiety

• Add context and enrichment logic to help AI triage

• Focus on recall and coverage rather than precision and volume

• Monitor AI triage performance and tune based on feedback

Time allocation shift:

• Less: Manual alert triage to validate detection quality

• More: Detection development, coverage expansion, AI tuning

For SOC Analysts

New workflow:

• Receive 20-30 pre-investigated cases per day instead of 200+ raw alerts

• Each case includes a full context gathered by AI

• Focus on judgment call and let the AI do the data gathering

• Provide feedback that improves AI over time

(updated section, thank yo,u Roger W. Roberts, for the feedback)

For Security Outcomes

Coverage improvement:

• From: 40-50% effective coverage (detect 80%, investigate 50%)

• To: 75-80% effective coverage (detect 80%, investigate 95%)

The Bottom Line

The shift from playbooks to agentic systems will be messy but inevitable. AI is pulling SOCs from static logic toward adaptive, self-improving systems. If the cloud era abstracted infrastructure, the AI era abstracts decision-making. Our processes must now teach machines how to operate within boundaries, not just describe what humans should do. That's not automation. That's architecture.

But here's the critical insight: This only works if you redesign your processes to take advantage of it.

Deploying broader detections into your current manual triage process just creates a bigger backlog. You need to transform how you handle the resulting alerts. That's where machine-executable investigation procedures come in.

Coming in Part 2: From PDF Playbooks to Machine-Executable Logic

Broader detection coverage only works if your investigation procedures can handle the volume. In Part 2, we'll explore the transformation that makes this possible:

You'll learn:

• Why your current SOPs don't work for AI (and what to do about it)

• How to convert human-readable playbooks into machine-executable logic

• A complete example: Suspicious login investigation (before and after)

• How does this transformation change the coverage funnel from 50% to 100% triage

• What this means operationally for your SOC team

The process tof ransformation is just as critical as technology. Get the SOP design wrong, and your AI will be making decisions based on incomplete or inconsistent logic. Get it right, and you unlock comprehensive coverage that was previously impossible.

Next week, we'll show you exactly how to do it.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

❗️News ❗️

I have been running the cybersec-automation blog for over two years now, and I decided it is time for a change.

So, I'm launching SecOps Unpacked, which is a sort of rebranding or simply expanding on what I was already doing.

While automation remains a core focus, I realised that security practitioners need more than just automation insights; they need independent research, practical frameworks, and honest vendor analysis.

I went with the idea of unpacking the security operations problems, exploring real solutions, and sharing what actually works in the field.

Some additions that you will notice are the Market research section as well tooling section (where I started a GitHub repo and soon will be sharing more Open Source tools)

I will keep my blog on the old domain until I figure out how to re-index everything and apply proper redirects.

Also, don't forget to check out the latest section, where I have a tracker of the AI SOC and Automation vendors (in-depth analysis coming soon)!

SecOps Unpacked - Research & Analysis for Security Practitioners

Unpacking SecOps problems. Exploring solutions. Sharing what actually works. Research-focused content for security practitioners and SOC engineers.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Based on what I've been hearing from the cybersecurity community, many of you are asking what the core capabilities of an AI SOC platform are that you should actually be looking at. A while back, I put together a blog that went through some of the high-level steps for implementation. This time, I want to deep-dive into the core capabilities as I see them.

You've probably seen my "AI SOC Shift" maps and graphics. They do a pretty good job of outlining the core pillars these platforms are targeting. It's important to see if they relate to your problems and to understand what type of platform you really need.

First, I’ll discuss the different implementation styles. Then, we’ll discuss data ingestion and what it looks like for AI SOC platforms. Finally, we'll cover the super important knowledge graph/DB. We'll also discuss investigations and, in the end, the response and feedback loop.

This edition is sponsored by Crogl

Built for analysts. Powered by your data. Private design

▪️Faster investigations - Cut Mean Time to Investigate (MTTI) by over 60%.
▪️Analyst-first design - Augments, never replaces. Built for real SOC workflows.
▪️Operational anywhere - Cloud, on-prem, and air-gapped environments.
▪️Customer-managed AI - Privacy, control, and compliance built in.
▪️Knowledge Engine - Automate triage, collects evidence, and documents everything.

CISO/SOC/MSSP pros, we’d love to meet you.

Implementation Styles

From what I’ve seen, there are a few types of implementations out there. We can start with the classic options like On-prem, Bring Your Own Cloud (BYOC), and SaaS. But what’s even more interesting is that some of these platforms now offer the option to bring your own LLM as well. There are some vendors moving into this space (check out the Vendor Spotlight section if you're interested).

I think this is going to be really relevant for large enterprises. Many of them are starting to develop their own internal LLMs and will want to use them. So, it’s a good sign that we're seeing vendors come up with capabilities that are truly enterprise-ready.

https://softwareanalyst.io/reports/ai-soc-industry-wide-report-2025

Data Ingestion

Here we have a few options as well. This is based on the vendors I’ve seen (demo, trial, or implemented), which is a bit over 30 in the AI SOC and next-gen security automation space.

First, you have the platforms that work just on top of your detections. These are the most common. They live on top of your detection layer, which can be a mix of SIEM, EDR, XDR, ITDR, Email Protection, and many others. What's important to consider here is what kind of tech they support. In my view, if it’s just about detection tech and excludes the SIEM, I want it to cover all my detection tools. Otherwise, you’re left with stuff you still need to handle manually or with your SOAR process.

For the platforms that also ingest from a SIEM, you should know that not all of them support every type of detection. This means many of the custom or unique detections you’ve built in your environment might not be supported, creating a gap.

Then there are others that live mainly on top of a SIEM or SOAR. This means you ingest everything you can into your SIEM, and the AI SOC platform takes any detection alert that comes out of it. The advantage here is that you can get really good coverage, as good as your SIEM allows. I've tried both implementations, and this one usually provides better results, assuming you can afford to get all your data into the SIEM.

One last thing here: some platforms can analyze your detections and come back with recommendations. Others will just ingest them and don’t put much effort into understanding why a rule runs in a specific way. For me, it's pretty important that the AI SOC platform tries to understand the logic of the rule, not just the alert it produces.

The Brains: Knowledge Graph/DB (Enrichment and Context)

This is a key component for any AI SOC solution. See it as the smart enrichment and knowledge automation engine that should go out and grab all the information it needs to understand an alert better.

From what I’ve seen, some platforms can connect to various tools that provide this information, from EDR, IAM, and Case Management to IaaS, Code Repos, and external threat intel feeds. Some might even give you some of that TI for free if you don’t want to bring your own.

I think this is critical for two reasons. First, to understand any alert, you need internal context. Think about whether the platform can ingest all the knowledge from your team, that means past incidents from Jira, wikis on Confluence or Notion, or documents on SharePoint. If you’re comfortable with it, see if they can even pull from specific IM channels where the team discusses cases.

Second, it's key to be able to connect to your internal case management solution like ServiceNow or Jira, or even GitHub repos. This allows the AI to check for relevant tickets or change requests related to the activity it's seeing.

So, put a lot of effort into evaluating this part. Make sure you can connect the tools you have so the platform can learn from your environment.

The Investigation Engine

This part can go into so many details that I'll probably do a follow-up blog on it. You can also check out a previous article where I walk through handling an EDR alert.

There are many different ways this is done, and many AI SOCs have their own method for stitching data together. But as far as capabilities, I’m looking for the following:

• Does it use response playbooks? I want a way to define custom scenarios or import my existing playbooks into the platform. That way, the platform is aware not just of best practices but also of my internal processes. In my own testing, I tried running it without any guidance from my playbooks, and the results weren't great. The investigation quality actually got worse over time.

• What's the right mix of guidance? I also tried adding my playbooks and letting the AI mainly follow that, and the results were good but a bit limited. We realized this would mean constantly updating our playbooks. The third approach, which got the best results, was to combine our internal playbooks with the platform's best practices. It understood our environment better and used all the log sources it needed to draw a conclusion. Plus, it took paths that were not in our playbooks but made sense in that specific context.

• Don't fall for the speed trap. These platforms mainly run API queries or search the SIEM. This means some investigations will only be as fast as your other tech. You have API rate limits and SIEM performance limitations. (Note: if your SIEM pricing is based on the number of queries you run, an AI will go a bit wild and run many more queries than a human would).

• Parallel vs. Sequential. It's also important to check if the platform runs actions in parallel or one after another. In security investigations, parallel actions don't always work because you often need to pivot based on what you find. Parallel makes sense for enrichment, but not for reactive threat hunting.

• Self-Verification. And one last point: does the AI verify and check itself once it completes an investigation? I've seen that when it does, it usually comes up with better outcomes.

• The Copilot. I can't forget the AI Copilot. At first, I thought it was mainly for starting a threat hunt, but I also want it available during an investigation. You should be able to pick up where the case was escalated and ask questions about specific steps or just continue the investigation yourself. It's important to make these interactions visible so all analysts can see what others have investigated. Good auditing and sharing options are key.

Taking Action: Remediation and Response

On this part, not many platforms are going all-in, mainly because response is harder to automate and it's something where you want more deterministic, predictable automation. From what I’ve seen, there are three types of capabilities:

1. Suggestions: They come up with response and remediation suggestions. This is the baseline I would expect. The key here is whether they understand your environment and suggest something that makes sense or just give you generic best-practice advice.

2. Basic Actions: These are the ones that have some simple response actions, like blocking an IP, suspecting a user session, resetting a password, doing an AI Bot interview with a user, alerting via IM, or quarantining a machine. I like this, as in some cases that’s all you need. It’s easy and convenient.

3. Native Automation: This is the third type, where platforms start to offer native automation capabilities like those in a SOAR platform, where you can build your own custom response workflows.

Closing the Loop: Lessons Learned

And the last part. Here we have a few capabilities to look for.

• Summaries & Reports: All platforms do alert and incident summaries; it’s like a core feature. But there are different types. Some allow you to create executive reports and give them a template for how it should look. You don't want to present reports to your management that are in a different style every time, so consistency is important. Others can do a summary of specific alerts you select and let you add artifacts to tailor it.

• The Feedback Loop: This is where things get interesting.

  • Some platforms give you no feedback mechanism. :)

  • Then you have the ones that offer basic suggestions, usually based on a summary report or dashboard.

  • Others come with suggestions on how to improve your detections. (This is my favorite,it’s what I want and I’ve even started to use it).

  • The ideal is a platform that gives you suggestions for both detection improvements and process improvements.

Final Thoughts

Evaluating an AI SOC platform isn't about finding a single killer feature. It's about understanding how all these different pieces, ingestion, knowledge, investigation, response, and feedback, work together. The goal isn't just to find a tool that closes alerts fast. It's to find a partner that can learn from your environment, adapt to your processes, and ultimately make your entire security program smarter. You need a system that thinks and learns with you, not just another black box.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

I’ve been going over a bunch of AI SOC implementations lately, and something hit me: control. It’s not just about having an “autonomous” system that investigates alerts. It’s about being able to see why it took a certain path, adjust the logic, and align it with your own environment. Without that, you end up with a black box that you can’t fully trust. 

This isn’t just theory. In almost every SOC I’ve worked with or advised, I’ve seen how fragile it becomes when you either (a) rely fully on vendor defaults, or (b) build out hundreds of breakable playbooks by hand. Both lead to pain in different ways.

This edition is sponsored by D3 Morpheus

Thinking AI SOC? Think Morpheus

Thinking AI SOC? Think Morpheus.
Morpheus delivers autonomy + control.
Built for Enterprise SOCs and top-tier MSSP/MDR.

▪️100% alert coverage
▪️ 95% alerts triaged in <2 mins.
▪️ Playbooks: autonomously built, user-adaptable 

24-7-365 autonomous investigations across any SOC stack, plus built-in case mgmt., IR flows, dashboards. 800+ integrations.

CISO/SOC/MSSP pros, we’d love to meet you.

Mapping the AI SOC Landscape

When I look at AI SOC platforms, I break them down into two ways.

First, on which stage of the incident lifecycle do they focus:

• Left - the data side: pipelines, log processing, detection engineering.

• Middle - enrichment, triage, and investigations.

• Right -response, remediation, and the feedback loop.

Some vendors specialize in one stage. Others stretch across multiple, but rarely all three.

Second, by implementation style:

• Plug-and-Play -quick to deploy, usually centered on the middle (triage, investigations). You connect data sources, and it starts producing outcomes with minimal setup.

• Build-and-Customize -more like next-gen security automation platforms. These let you create workflows from scratch, wire them to alerts, or even run “headless” background automations. They usually cover the middle and right, sometimes the left as well.

Over the last 2–3 years, most new AI SOC startups have landed in the middle with plug-and-play products. Security automation platforms lean toward build-and-customize.

Both approaches have tradeoffs, and understanding them is key.

The “Build-and-Customize” Trap

Build-and-customize platforms feel like classic real-time strategy games. You start with nothing and design everything yourself: the integrations, the escalation logic, the deterministic flows. You’re in charge.

That control is great, but it also means you own the complexity.

Here’s the trap: deterministic automation doesn’t scale well if you try to build one workflow for every single detection use case. In one SOC I worked with, the team had over 200 automations, each tied to a specific detection. The result? They needed more engineers than analysts just to keep workflows alive. Every time an API changed, a field was renamed, or a vendor shifted its schema, half the playbooks broke.

So while this model is powerful, it quickly becomes breakable. It works well if you’re building “macro” workflows (like log ingestion or enrichment pipelines). But if you try to encode every micro-decision into a deterministic playbook, you end up with a maintenance nightmare.

The “Plug-and-Play” Black Box

Plug-and-play AI SOCs are the opposite. Think of them as strategy RPGs: the world is pre-built, the rules are set, and you just play the role you’re given.

The upside is obvious: fast time-to-value. You connect your SIEM or log sources, and suddenly the platform is triaging, clustering, or even investigating alerts for you. For many orgs that are short-staffed, that’s appealing.

The downside is just as obvious: no visibility into how the logic works, and no way to adjust it.

• What if the platform suppresses something you consider critical?

• What if its default enrichment path doesn’t fit your environment?

• What if you just want to add one custom validation step before remediation?

In most cases, you can’t. You’re locked into the vendor’s logic. And when your team doesn’t understand or can’t shape the process, trust in the tool drops fast.

Where Automation Already Works Well

Before talking about hybrids, let’s ground this in where automation already delivers value. From my own experience, the biggest wins are on the left side of the lifecycle.

Take log ingestion. Many SaaS tools don’t provide streaming log integrations. You either need to pull data via API or pay for a third-party connector. I’ve built automations that:

• Fetch audit logs on a schedule,

• Parse and normalize them,

• Push them into S3 or another bucket,

• Then feed them to a SIEM in the format it expects.

The benefit is twofold: the SIEM gets data in a clean, expected schema, and you offload parsing/storage from the SIEM, which saves costs.

In detection engineering, I’ve built workflows that enrich threat intel feeds, extract TTPs, and generate hypotheses for new detections. For example, if intel shows a threat actor shifting to a new initial access vector, the automation surfaces that and suggests where detection coverage may be missing.

There are also operational automations:

• Creating backlog tickets automatically when a false positive is reported.

• Triggering attack simulations when new detections are deployed.

• Sending requests to red teams to validate coverage through adversary emulation.

These aren’t glamorous, but they save huge amounts of manual effort.

The Messy Middle

The middle of the SOC lifecycle isn’t just “messy;” it’s where the real detective work happens. You’ve pulled in enrichment and gathered context, and now you need to connect the dots into a coherent incident story.

This is harder than it looks. Evidence isn’t uniform. Sometimes you’re pulling infrastructure data and need to cross-check with change requests or asset management logs. Sometimes it’s endpoint behavior, where there is no clear baseline, users run all sorts of random processes on their machines, and you need to know what’s “normal” in this business context. Other times, you’re correlating identity data, asking whether a login pattern aligns with known behavior for that role, that region, or that application.

In practice, analysts spend much of their time bouncing between SIEM queries, log sources, and business systems, running searches and pulling fragments of context. The challenge is less about gathering raw data and more about stitching it into a narrative that makes sense. That’s why the middle ground has historically been so fragile for automation; you can’t just script it once and call it done. The process changes with every environment, every investigation, every new clue.

This is where AI could be transformative, if it helps analysts assemble evidence and propose connections, but still shows why it drew those links. Without that transparency, it’s just guessing in the dark.

A Hybrid Model: Autonomy with Guardrails

So instead of being stuck with a black-box tool or a maintenance nightmare, imagine something in the middle. A hybrid model is all about getting the best of both worlds: the speed and scale of AI automation, with the control and flexibility your team actually needs.

Here’s how I see it working: Instead of coding every playbook by hand, you just drop an alert into the AI SOC platform. From there, it generates a full investigation workflow on the fly.

But here's the key part, it's not a black box. You can see all the steps it plans to take, and you can fine-tune them as needed. You could even upload one of your existing runbooks from Confluence, and the platform would use it as a template to shape its automation logic.

I like to think of it with a gaming analogy.

• The "City-Builder" Phase: This is the build-and-customize part where you lay down the rules. You set up the integrations, define your critical assets, and build deterministic guardrails, like "for any action on a domain controller, you must get human approval".

• The "RPG" Phase: This is where the autonomous AI agents operate within the world you just built. They can investigate alerts, enrich data, and even suggest remediation, but they always have to follow the rules and stay on the roads you created.

This approach combines the strengths of both copilots and fully autonomous agents. Key capabilities usually include:

Balanced Autonomy: The system handles the routine, high-volume stuff on its own but knows when to stop and escalate tricky or high-impact decisions to a human analyst.

Flexible Exploration: Your analysts aren't locked into a rigid workflow. They can pivot from the AI's automated findings and start asking their own questions in an interactive chat, letting them dig deeper whenever they need to.

Customizable Logic: The AI's workflows can be tailored to fit your SOC’s specific needs, giving you a good balance between automated consistency and the flexibility to handle unique threats.

Ultimately, this balance, autonomy with guardrails, is what will make AI SOCs something we can actually rely on. It gives you a system that can handle the massive scale of alerts while making sure a human is still in the driver's seat for the decisions that really matter.

Final Thoughts

Look, at the end of the day, there's no magic answer here. It’s super important to figure out what fits your own environment. Both the plug-and-play and the build-it-yourself platforms have their place.

If you're short on people and just need something running fast, a plug-and-play tool is tempting. But you're stuck in their world, using their logic, and it's basically a black box. On the other hand, if you have a big engineering team, maybe building everything from scratch sounds good. But I've seen how that turns into a maintenance nightmare that costs a fortune to keep running.

This is where the hybrid model has some serious advantages. It's way faster to deploy than trying to build everything custom from the ground up, since the core AI investigation logic is already there. The cost of maintenance is also a lot lower because you’re not trying to keep hundreds of automation playbooks alive. And you still get the flexibility to tweak the workflows and make sure the system operates in a way you can actually trust. You get the speed of AI without having to give up all the control.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Audio Version

I’ll take some responsibility here. Maybe I started this movement, maybe not, but I’ve definitely been pushing it for a while. So now I’ll give it a proper name and see if I get some credit for it. In the end, it’s not about who said it first. It’s about sharing knowledge and helping the community make sense of what’s happening.

It's been a couple of years now since AI really started hitting the cyber field, especially in Security Operations. We started with AI copilots, then the focus moved to the sweet spot, investigations, and now we're seeing a push to "shift left" and "shift right."

And if you’re wondering what I mean by shifting left or right in this context, let’s clear that up. Unlike the messy attempts of “shift left” in SDLC, here it’s simple: shifting left or right means asking which stage of the IR cycle you are applying AI to. For me, that’s the best way to evaluate, identify, and understand AI’s true capabilities for SecOps.

So let’s give this framework a name: The SecOps AI Shift Map.

This edition is sponsored by Exaforce

From Zero to AI-Driven SOC

Exaforce is a breakthrough AI SOC platform that infuses AI into every stage of your SOC lifecycle, across detection, triage, investigation, and response. We help you reduce your manual effort and improve your outcomes, while driving down costs. To learn how Exaforce is helping organizations like yours, book a call with us or join the upcoming webinar.

Shift Left: The Land of Detections and Data

If you break down the IR cycle into three stages, left, middle, and right, you can start mapping where AI is actually being used. On the left side, we have data ingestion, log processing, and detection engineering. Think of it as the foundation: we set up controls like EDR, network protection, IAM, cloud guardrails, email security, the usual stack. 

Then comes logging. You centralize everything in a SIEM, prioritize sources, and slowly build toward the impossible dream of “100% visibility.” 

After that comes detection engineering: choosing the right log sources, building threat profiles, and designing detections based on expected TTPs. That’s the left side of the house, and it ends with a detection firing.

The Middle: The Investigation Sweet Spot

The middle side is where things get interesting, and honestly, where most of the industry is today. Once a detection fires, you need procedures. Call them SOPs, runbooks, or playbooks, they all serve the same purpose: guiding what you need to look at. Here comes enrichment, and I know there are strong opinions on whether this should sit at the detection layer or the investigation layer. In reality, it depends. Some enrichments definitely make detections stronger, but many, like CTI lookups, alert similarity, correlation, or even tribal knowledge, make more sense during investigation.

Once enriched, the investigation starts. I like to map this stage to the 5Ws: Who, What, When, Where, and Why. Answering those questions leads you to a conclusion, a verdict.

We have a few types of verdicts:

• True Positive: This is malicious; we have a compromise or an issue that needs to be fixed.

• Benign (False Positive): This alert shouldn't have triggered, and the detection needs to be fine-tuned.

• Benign (True Positive): The alert triggered as expected, and the activity was confirmed to be normal. (Technically, some folks call this a "Benign True Positive" since the alert did fire correctly, but the point is the same: the activity was expected and not a threat.)

Note that we are measuring metrics on alerts here, not incidents. Incidents are a different bucket. They can start directly in the investigation phase without an alert, and that’s where we might find a False Negative, no alert was generated, but someone (an employee or an external party) identified a compromise.

Shift Right: Remediation, Recovery, and Broken Feedback Loops

And now for the right side of the house. This is where remediation, recovery, and lessons learned happen. It’s also the hardest part to automate or apply a lot of AI to, especially remediation.

You’ll hear many people say this is where SOAR failed. I’d say no, it didn’t, we failed here because our processes are a mess. True automation hits a wall when it runs into a lack of API coverage for critical internal tools, undocumented tribal knowledge, a culture of risk aversion, and of course, the dreaded change management process. Remediation isn't always straightforward. The larger and more regulated the organization, the more you run into these roadblocks. You can’t just go wild and do what you want. No one wants to give a service account to a SOAR or AI SOC solution that has full write access. Even if some actions are pre-approved, the risk is just too high.

The idea is that you don't have many compromises, so you shouldn't need to run these actions often. This beats the purpose of building automations that run only a few times a year. (And if you're an org that needs to run these actions often, I think you should go back to the left side and fix your issues there).

Recovery is a bit easier to automate, and this is where IT and DevOps automation usually comes into play. And lessons learned? This is where AI shines. It’s great at writing a summary, getting you an executive report, and formatting it nicely. The part we need to improve is how we feed this information all the way back to the left side to constantly get better. We’re not doing a great job there. Where we fall short is feeding those lessons back to the left side to improve detections and logging. And let’s be honest: for many vendors, fewer problems don’t translate into revenue, so the feedback loop rarely gets prioritized.

So, Why Did Everyone Start in the Middle? (The Cake Analogy)

When AI for SecOps first showed up, vendors went straight for the middle, the investigation. It’s like getting a fancy cake. You ignore the heavy fondant on the outside (the broken detections) and go for the sweet, sugary center. It gives you a quick sugar rush (hundreds of alerts closed fast!), and you feel great.

(Weird analogy, I know).

But you're not fixing the real problems. The fondant still tastes bad, and the sugar crash is coming. The "lessons learned" are that you ate too much cake, but a week later, you forget and just remember the sweet part. You want that good feeling of eating the sweet middle part (all those 100 alerts closed fast).

So yeah, a long way of saying it started in the middle because it was the easiest part. It's the main area where we don't need as many deterministic things, and we can easily use GenAI to do analysis and come up with a verdict.

Beyond the Investigation: Fixing the Whole Problem

But now we're realizing that just doing the investigation isn't enough. What about our broken or incomplete detections? What about the log sources that are missing key logs? What about the simple response actions?

That's what we need to actually fix the problem. I want to get recommendations on which detections should be fixed and which log sources should be improved. I want my AI to be able to reach out to a user and ask for more information if needed, and then quarantine their machine and reset their password. 

Simple, right?

Sad story is, not many are doing it yet.

Final Thoughts

The SecOps AI Shift Map is a way to frame where AI is being applied across the IR cycle. It helps us evaluate vendors, set expectations, and identify gaps.

We started in the middle, because it was easy and satisfying. But the real progress ,the real SOC of the future ,is in shifting left and right. That’s how we’ll build AI SOCs that don’t just close alerts faster, but actually make security operations stronger end to end.

Just before I jump to the vendor spotlight, I wanted to reassure you, the audience: all the vendors I mention here are ones I've had a demo with to understand their capabilities. It’s not just for the sake of having a vendor highlight; it's something I've seen and evaluated, and I'm giving you my insight on what they're good at.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Memory makes GenAI useful in the SOC and also makes it opinionated. That opinion can drift. Some stacks become optimistic and default to benign. Others become pessimistic and default to malicious. Both are forms of bias. You can control this with explicit objectives, memory hygiene, calibration, and drift monitoring. Citations and a practical checklist are below.

Table of Contents

• The observation

• Where the bias actually comes from

• Monitoring Drift in AI SOCs

  • Population Stability Index (PSI)

  • ADWIN (Adaptive Windowing)

  • Why this matters in practice

• Why AI SOC Drift Is Different From Traditional Mod …

• How to Control Optimism vs Pessimism on Purpose

  • 1. State your loss function

  • 2. Separate facts from verdicts in memory

  • 3. Enforce memory hygiene

  • 4. Use structured, two-pass reasoning

  • 5. Calibrate regularly

  • 6. Monitor drift continuously

  • 7. Test with counterfactuals and red teams

• What to ask your vendor

• What we know from the literature and industry

• References

• Closing view

The observation

In a clean world, AI investigations end in one of three outcomes: benign, suspicious, or malicious. In the field, once you add memory, the model starts anchoring to prior cases and narratives. Over time, I see two failure modes:

• Optimistic drift: the AI looks for evidence that activity is benign and closes too fast.

• Pessimistic drift: the AI assumes breach and marks too many items as malicious, flooding triage.

Both are biased patterns reinforced by memory. This is not “AI gone wrong.” It is a predictable effect of feedback, sampling, and incentives. Anchoring, confirmation, and availability biases show up in ML pipelines just as they do in humans.

My take: I prefer a pessimistic default in security. Assume compromise and plan for worst case, then reduce noise with controls rather than miss a critical event.

Do you enjoy CyberSec Automation Blog Content?

Nominate for SANS Difference Makers Awards

The SANS Difference Maker Awards are open for nominations, and I’d love your support. I’m applying in the Media Creator of the Year category for my work on CyberSec Automation. If you like the content I’ve been sharing, it would mean a lot if you could take a minute to nominate.

Where the bias actually comes from

• Anchoring via memory

  Retrieval of previous conclusions, tickets, and notes can anchor the model’s current reasoning. If memory stores verdicts, not only facts, you increase confirmation bias. NIST and ACM both call out bias as socio-technical, not just model-level.

• Label/feedback loops

  If analysts reward “fast closes,” your reinforcement signal pushes optimism. If you reward “catch anything suspicious,” you push pessimism.

• Dataset shift and model drift

  Your traffic, tools, and attacker mix change. That is covariate shift and concept drift. You must detect and respond to it. Practical detectors include PSI for distribution shift and streaming detectors like ADWIN.

• Calibration decay

  Confidence scores stop matching reality as the environment changes. Track Expected Calibration Error and similar metrics to keep “probability of malicious” honest.

• Domain specifics of cyber

  In cybersecurity, bias does not only waste time. It can either hide attacks (optimistic) or burn out analysts and suppress signal (pessimistic). Industry pieces echo this tension in practice.

Monitoring Drift in AI SOCs

Two practical techniques you’ll hear about in ML Ops — and that are directly useful in AI SOC — are PSI and ADWIN. Both are ways of spotting when your model has started to “see the world differently” than it did at training or deployment time.

Population Stability Index (PSI)

• PSI is a simple way to measure whether the distribution of features (inputs the model uses) has shifted.

• For example: imagine your model relies heavily on login geolocation or file hash rarity. If the frequency distribution of those features changes a lot compared to your baseline (say, suddenly 30% of logins are from a new region), your model is now making predictions on data it hasn’t really been trained for.

• We typically set thresholds:

  • PSI < 0.1 → stable (no real change)

  • 0.1–0.2 → moderate shift (keep an eye on it)

  • >0.2 → warning level, drift may be affecting results

  • >0.25 → action required (retrain or re-evaluate)

• In the SOC, you’d use PSI on high-value features like source IP reputation scores, authentication method, or endpoint process ancestry — the signals that drive most of your verdicts.

ADWIN (Adaptive Windowing)

• ADWIN is a streaming drift detector. It looks at incoming telemetry in real time and detects if the statistical properties of the data have changed.

• Think of it as a moving window: if the recent data looks very different from the older data, ADWIN flags drift.

• Example in a SOC:

  • You’re monitoring failed login attempts per user per hour. Normally the distribution is steady. Suddenly, in the last hour, the rate jumps in a way that doesn’t fit past behavior. ADWIN detects this as a distribution change — signaling that the model’s prior assumptions may no longer hold.

• ADWIN is valuable when your SOC ingests continuous, fast-changing data (auth logs, endpoint events, netflow).

Why this matters in practice

• PSI tells you when your AI is drifting because the population of data has shifted.

• ADWIN tells you when your AI is drifting because the stream of data is behaving differently in real time.

Both should trigger drift alerts in your SOC platform. When thresholds are hit, you either retrain the model, adjust decision thresholds, or at least shadow-test the model to confirm it’s still making reliable calls.

Why AI SOC Drift Is Different From Traditional Model Drift

There’s something specific to cybersecurity worth calling out.

When you roll out an AI SOC platform, you’ll either:

• Feed it historical data (if the platform supports it), or

• Let it start fresh, learning in shadow mode from day one of deployment.

In the traditional SOC workflow, when a human analyst processes an alert, the output is often a one-liner:

• “False positive because xyz.”

• “Normal activity, change request attached.”

That limited context doesn’t feed the model much bias. It tells the system how the case ended but doesn’t provide a lot of rich narrative that can anchor future decisions.

But with AI handling investigations, the story changes. Instead of one-liners, you get one, two, sometimes three paragraphs of reasoning explaining why an alert was closed as benign, suspicious, or malicious.

And here’s the kicker: most AI SOC platforms are built so that analysts approve or deny those AI conclusions. That means the model isn’t just learning the verdict — it’s learning from the entire summary and narrative it generated.

The result?

• The more alerts you approve with “benign” narratives, the more the model drifts optimistic.

• The more you approve “malicious” narratives, the more it drifts pessimistic.

• And beyond optimism/pessimism, the model starts encoding other forms of bias hidden in the text — anchoring to particular arguments, data sources, or analyst preferences.

This makes AI SOC drift qualitatively different from classical ML drift. You’re not just feeding it labels. You’re feeding it reinforced narratives — which are far more context-rich and far more prone to anchoring.

How to Control Optimism vs Pessimism on Purpose

1. State your loss function

Every SOC has to decide which mistake is more costly:

• False Negative (FN) → Missing an intrusion.

• False Positive (FP) → Investigating noise.

In most environments, FN > FP — a breach is worse than wasted cycles. But not all teams weigh it the same:

• A resource-constrained SOC may set stricter limits on FPs to avoid burnout.

• A high-risk sector (finance, healthcare) will tolerate more noise to minimize missed threats.

The key is to make this explicit. Don’t let the model’s default bias define your risk appetite. Encode the loss function into:

• Thresholds: e.g., “AI must be 85% confident to close as benign, but only 60% confident to escalate as malicious.”

• Routing rules: e.g., suspicious verdicts always go to Tier 1, but any high-risk suspicious (based on threat intel correlation) goes straight to IR.

Treat this as SOC policy, not a hidden “prompt hack.”

2. Separate facts from verdicts in memory

One of the biggest sources of anchoring bias is how memory retrieves past cases.

• If the AI sees a verdict like “benign – normal user behavior” in memory, it may bias its current conclusion toward benign.

• If memory only retrieves facts (e.g., “User X logged in from IP Y at 03:00, MFA success”), the AI can evaluate evidence without inheriting the past verdict.

Practical controls:

• Store artifacts, signals, and context → log snippets, process trees, enrichment.

• Down-weight final labels → allow retrieval of verdicts, but apply less importance than raw evidence.

• Force “evidence-first” prompts → e.g., “Summarize the facts before giving a conclusion.”

This keeps memory as a knowledge base, not a verdict repeater.

3. Enforce memory hygiene

Think of memory like a SIEM data lake — garbage in, garbage out. Without hygiene, bias compounds.

• TTL (Time to Live): Don’t let outdated conclusions anchor current cases. E.g., verdicts expire after 30 days unless reaffirmed.

• Source tags & provenance: Every memory chunk should record its origin — log type, analyst name, AI agent version. This makes retrieval explainable.

• Retrieval filters: Prefer multi-source evidence. For example, if two independent log sources (e.g., auth + EDR) align, rank that memory higher than a single noisy source.

This reduces toxic bias accumulation and prevents “zombie verdicts” from skewing current reasoning.

4. Use structured, two-pass reasoning

Humans avoid confirmation bias by debating, and AI needs the same. A two-pass system creates an internal “red team.”

• Pass A: Build a case file of observations and hypotheses. E.g., “Unusual PowerShell execution observed, correlated with new registry keys.”

• Pass B: Adversarial review. AI (or a secondary agent) argues the opposite verdict. E.g., “This could also be normal IT admin activity — prior change tickets show similar actions.”

The final verdict must resolve both arguments. This mirrors human analyst workflows (peer review, escalation) and makes conclusions more balanced.

5. Calibrate regularly

Raw confidence scores are almost always misleading. Calibration ensures “80% confidence” really means “8 out of 10 times correct.”

• Metrics:

  • ECE (Expected Calibration Error): measures mismatch between predicted vs actual accuracy.

  • Brier score: penalizes both overconfidence and underconfidence.

• Operationalization:

  • Re-tune thresholds if calibration drifts.

  • Publish a monthly calibration report to SOC leadership showing whether the AI’s confidence is still trustworthy.

Well-calibrated AI allows you to set rational escalation policies instead of “gut feel” thresholds.

6. Monitor drift continuously

Data never stays static. Model drift is inevitable. The only question is whether you catch it.

• PSI (Population Stability Index):

  • Compares historical feature distributions vs live data.

  • E.g., login location mix changes drastically.

  • Thresholds: >0.2 = warning, >0.25 = action.

• ADWIN (Adaptive Windowing):

  • Monitors real-time streams for sudden changes.

  • E.g., spike in failed logins per user compared to historical patterns.

• Response:

  • Trigger shadow evaluation, partial retraining, or force suspicious verdicts when drift is detected.

  • Automate drift alerts into the SOC dashboard (same way we alert on log ingestion failures).

7. Test with counterfactuals and red teams

SOC AI should be tested like detections: continuously and adversarially.

• Counterfactuals: Hold out known attack scenarios and near-miss benigns. Evaluate AI weekly against this set.

• Rotation: Refresh test data monthly to avoid overfitting.

• Red teaming: Actively simulate edge cases — “what if MFA fails but user behavior is normal?” — to stress-test reasoning.

NIST AI RMF highlights this: continuous evaluation tied to business harm, not just technical accuracy.

What to ask your vendor

• How do you prevent anchoring from prior verdicts when using memory or case history?

• Do you provide calibration reports and can we export ECE or similar?

• What drift detectors are built-in (PSI, ADWIN, custom) and how are alerts surfaced?

• Can we tune the loss function or cost ratios for FN vs FP by use case?

• Do you support two-agent review or adversarial reasoning before final verdict?

• How do you version prompts, memories, and model configs so we can audit changes?

• Show us your approach aligned with NIST AI RMF controls for measurement and monitoring.

What we know from the literature and industry

• Bias can enter at data, model, and deployment stages. You must treat this as a socio-technical problem, not just tuning a model.

• Cybersecurity is already seeing bias outcomes that either hide threats or inflate noise. Leaders are concerned and are asking for measurable controls.

• Drift is normal in live systems. Use PSI for distribution monitoring and ADWIN for streaming change detection. Build playbooks that trigger re-evaluation when these fire.

Confidence must be calibrated if you expect analysts to trust scores. Track ECE and retrain or re-threshold when it degrades.

References

• ACM: Biases in AI Systems

  https://cacm.acm.org/practice/biases-in-ai-systems/

• Interface Media: Exploring the Impact of AI Bias on Cybersecurity

  https://interface.media/blog/2024/12/24/exploring-the-impact-of-ai-bias-on-cybersecurity/

• Chief Executive: Why AI Bias Is a Growing Cybersecurity Concern

  https://chiefexecutive.net/why-ai-bias-is-a-growing-cybersecurity-concern/

• NIST AI Risk Management Framework (AI RMF 1.0)

  https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf

• Concept Drift Detection in Data Streams (ADWIN, Gama et al.)

  https://homes.di.unimi.it/~cesabian/Pubblicazioni/mlj10.pdf

• Measuring Calibration in Deep Learning (Guo et al., 2017)

  https://arxiv.org/abs/1706.04599

• Population Stability Index (PSI) for Monitoring Model Drift (Practical Guide)

  https://docs.snowflake.com/en/user-guide/mlops-psi

• NIST: Bias in Artificial Intelligence

  https://www.nist.gov/news-events/news/2022/03/nist-study-evaluates-tools-mitigate-bias-artificial-intelligence

Closing view

I still prefer a pessimistic default in security. Assume breach. Pay the cost of extra triage while you mature memory, calibration, and drift controls. You can dial back noise with evidence requirements and better calibration. You cannot easily recover from a missed intrusion.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Here's another edition of the CyberSec Automation blog, and you're probably noticing it looks a little different. I recently launched the CyberSec Automation Interview Series / Podcast, and I wanted to share some of the recent episodes with you.

In these sessions, I chat with founders of cybersecurity companies, especially in the SecOps space, and other practitioners. The idea is to have a casual, no-BS conversation about what's really happening in security automation. Events are streamed live on LinkedIn, and I've also created a YouTube channel where you can catch up on past episodes. Go ahead and subscribe if you want to get notified about new ones.

The episodes are usually tied to a blog post, making it easier to follow the topic. Some are vendor-agnostic, while in others, founders explain how they're tackling a specific problem with their solution.

This monthly edition of the blog will be all about the podcasts I host and the ones where I'm a guest.

Since this is the first one, here are the episodes and the related blogs.

When Does It Make Sense to Automate? When Does AI SOC Actually Work?

Join us for the next episode of the CyberSec Automation Interview Series!

In this episode, I sit down with security pros Andrei Cotaie and Cristian Miron to tackle a big question in modern SOCs: when does it make sense to automate, and when should we trust AI in the SOC?

We'll talk about the three main stages of the SOC workflow: detection and log ingestion, investigation, and response. We'll get into where automation provides the most bang for your buck, where AI SOC is starting to make a real impact, and where it’s still too early to rely on it.

It's just a laid-back chat with real-world examples of how we're implementing and using AI SOC and automation.

You can check out the upcoming event here: https://www.linkedin.com/events/whendoesitmakesensetoautomate-w7360612707361841152/

And the related blog post is here:

Why SOC Analysts Ignore Your Playbooks

Uncover why SOC analysts resist traditional playbooks and learn how to create more effective, actionable security automation strategies that actually work.

Smashing the Myth of the Single Pane of Glass

I sat down with Senad Aruc, CEO of Imperum, to deconstruct the "single pane of glass" architecture.

We got pretty technical, talking about why simple LLM wrappers don't work without native Threat Detection, Investigation, and Response (TDIR), the future of SIEM, how to deal with API limitations, and the kind of stack you need for a real Autonomous SOC.

Watch the full episode here:

And here's the blog post that goes with it:

Stop Chasing the Single Pane of Glass

Debunk the "single pane of glass" myth in cybersecurity: Explore why chasing unified platforms falls short and discover smarter approaches to security monitoring and response.

Building AI for SOCs That Analysts Don’t Hate

In this episode, Tom Findling from Conifers.ai joins me to talk about what it takes to build an AI-driven SOC platform that analysts actually want to use.

We dive into the key features every AI SOC platform should have, best practices for implementation, and how to set realistic expectations from the get-go. No hype, just practical advice.

Check out the conversation here:

The related blog is:

Automate Smarter, Not Louder: Using Interactive AI Feedback Loops

Revolutionize SecOps with AI-driven feedback loops: Learn how smart automation transforms threat intelligence, detection, and response strategies effectively.

Where I Was a Guest

I also had the chance to be a guest on a couple of other podcasts.

EP 6. From Data Chaos to Detection Engineering: How to Automate What Really Matters in the SOC

I joined Balázs Scheidler on the Data Strikes Back podcast to talk about a topic that's close to my heart: your SOC isn’t failing because of bad detections; it's failing because your data is a mess.

We talked about:

• How to use automation for more than just incident response, like in log management and data pipelines.

• Why "detection as code" is a waste of time without good schema discipline.

• How to handle massive amounts of legacy syslog data (we're talking 50TB/day) without blowing your SIEM budget.

• When it makes sense to standardize, transform, or just automate everything.

You can listen to it here: Data Strikes Back on Spotify 

Defender Fridays by LimaCharlie

I was also invited to join an episode of Defender Fridays by LimaCharlie. It was a great conversation about the current state of security automation and where things are headed.

You can watch that here: Defender Fridays