Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

1. Introduction

The AI SOC market is growing fast and there are products on it that are doing serious work. Some of them have strong integration capabilities, solid reasoning engines, and response actions that actually execute in production. The market has come a long way in four years.

But there is a problem with how we evaluate these products.

When every vendor says "AI-powered response," that phrase covers everything from a fully autonomous isolation workflow to a chatbot that suggests you maybe think about resetting a password. Both get the same label in the marketing material. Both show up in the same analyst reports. And when a security team sits down to compare three products, they have no standardized way to measure the gap between "our AI handles response" and what that actually means in operational terms.

Some products are close to real autonomy in specific domains. Some are strong in analysis but thin on execution. Some have broad coverage but almost nothing runs without human approval. These are all valid positions on a maturity spectrum. The problem is that there is no shared framework to place them on that spectrum consistently.

So we built one.

We call it ARMM. And yes, the name is intentional.

A decade ago, the SOAR generation solved half the problem. We built the arms. Playbooks, integrations, automated response workflows. The execution layer was there. What was missing was the brain. Every decision tree was hand-coded. Every branching logic was written by an engineer who had to anticipate every possible scenario. The arms moved, but only along rails that humans laid down manually. When the scenario deviated from the playbook, the arm froze.

Now the AI SOC generation has solved the other half. We built the brain. LLMs reason across alerts, correlate context, analyze logs, and make judgment calls that no static playbook could replicate. But somewhere along the way, a lot of products forgot to attach the arms. The reasoning is strong. The analysis is sharp. And then it hands you a summary and says "here is what you should probably do." The brain thinks. The arm does not move.

ARMM evaluates both. The reasoning quality, the decision-making maturity, the trust you can place in the AI's judgment. And the response capability, the execution depth, the ability to actually take action without three humans supervising. It weighs the arm heavier because that is where the industry gap is widest right now. But it does not ignore the brain, because an arm without a brain is just a SOAR playbook and we already know how that story ended.

ARMM is a structured scoring system for evaluating what an AI SOC solution can actually do in the response layer. It covers 80+ response capabilities across six domains: Identity, Network, Endpoint, Cloud, SaaS, and General Options. And it provides a common language so that when someone says "we handle response," there is a way to ask: at what level, across how many actions, and with what degree of autonomy?

The CyberSec Automation Blog has published over a dozen articles and podcast episodes covering what makes a good automation program succeed, how to evaluate tools, and how to structure decision-making around security automation purchases. We have built tool comparison lists, evaluation checklists, and decision frameworks. ARMM is the next step in that work.

2. Why Another Framework

Most existing evaluation methods for AI SOC solutions are either vendor-produced (and therefore biased toward their own capabilities) or too generic to capture the specific nuances of AI-driven response. Analyst reports compare products at a feature-list level without measuring automation depth. Vendor demos show best-case scenarios without exposing the operational friction underneath.

Our focus is narrow and deliberate: response capabilities. Most AI SOC solutions already deliver strong reporting and analysis features. They can summarize alerts, correlate indicators, and reduce false negatives in a mature environment (we emphasize mature because these solutions need access to quality logs and, in more advanced implementations, to organizational documentation and environment-specific context). Where the industry needs structured evaluation is in the response layer: the actions an AI SOC solution can take, how autonomously it can take them, and under what conditions.

We acknowledge that some of the capabilities listed in this framework may seem aspirational at this stage. That is by design. The framework is intended to serve both as a current-state evaluation tool and as a forward-looking roadmap.

We are not scoring specific vendors. The goal is to establish a shared methodology that allows security teams to answer questions such as:

• Which solution provides more relevant response capabilities for my environment?

• Which solution operates at a higher level of autonomy for the actions that matter to my program?

• Which solution can help me reduce my alert backlog without requiring additional headcount?

For product managers working on AI SOC products, the framework serves as a competitive analysis baseline:

• Where is my competition positioned, and what capabilities are driving their wins?

• What high-value capabilities are underserved across the market?

• Am I investing engineering resources in features that security practitioners actually prioritize?

Because this is a fast-moving space, we are starting at version 0.1. This is a living document. Version 1.0 will be designated when the framework reaches a level of stability and community validation that warrants it.

3. Scoring Methodology

ARMM supports two distinct approaches to scoring, each designed for a different operational question.

Evaluator Mode is the straightforward path. You score each capability on the 0-1-2 scale described above (with the 1C, 1G, 1A sub-levels) and the framework calculates your coverage rate, automation depth, and per-plane breakdown. The tier placements come from ARMM's reference tables. You do not need to factor in your organizational context. This mode answers one question: given two or more AI SOC products, which one covers more of what I need and at what automation level? It is built for procurement teams, SOC managers running vendor evaluations, and anyone who needs a side-by-side comparison without spending weeks on it.

Builder Mode adds a second scoring layer on top. Instead of relying on fixed reference tiers, you score each action across three axes: Trust (how much confidence does your implementation warrant), Complexity (how hard is it for your specific team to build and maintain), and Impact (what is the blast radius if something goes wrong). The action score becomes T + C + I, and the tier placement shifts based on your organizational reality. The same action that scores Entry for a mature team with established automation pipelines might score Explorer for a team that is deploying its first AI SOC integration. This mode answers a different question: given my team, my environment, and my risk tolerance, where should I invest engineering effort to move up the maturity ladder? It is built for product managers, engineering leads, and internal SOC teams running their own automation programs.

Both modes evaluate the same six planes and the same 80+ response capabilities. Both produce per-plane breakdowns and a composite maturity label. The difference is whether you want a product-level comparison (Evaluator) or an environment-aware implementation roadmap (Builder). The public ARMM app at armm.secops-unpacked.ai  supports both.

3.1 The Capability Scoring System (0-1-2)

Each response capability in the framework is scored on a three-level scale that measures the degree of automation available:

0 (Not Available): The feature does not exist in the product. There is no mechanism, manual or automated, to perform this action through the AI SOC solution.

1 (Available with Human Involvement): The feature exists but requires some form of human interaction before execution. Because human involvement can range from full collaboration to a simple approval click, this level is subdivided into three sub-categories:

• 1C (Collaborator): The solution requires continuous back-and-forth interaction with an analyst to reach a response action. The AI acts as a partner, not an autonomous agent.

• 1G (Guide): The solution generates a plan and presents options for a specific action, but it is not confident in recommending a single path. It lays out alternatives and lets the analyst choose.

• 1A (Approver): The action is essentially ready to execute. The AI has determined the correct response and prepared the action, but requires a human to click approve before it fires. This is the closest step to full automation while still keeping a human in the loop.

2 (Fully Automated): The action is performed without any human involvement. The vendor (or internal implementation) has demonstrated that the AI SOC solution can execute this action with sufficient confidence that no human review is required. At the time of writing, level 2 is exceptionally rare for most response categories. The framework includes it to establish the target state and to differentiate products that are moving in that direction from those that are not.

3.2 The Three Scoring Axes (Builder Mode)

In Builder Mode, each response action is evaluated across three dimensions:

Axis 1: Decision Fidelity and Programmatic Trust (T)

This axis measures the confidence level warranted by the AI SOC implementation. It correlates directly with implementation quality: reasoning log depth, context-aware decision-making, and guardrails against hallucination.

• T = 1 (Enrichment): AI output assists human-led investigations. The AI provides context and data but does not recommend or execute actions.

• T = 2 (Validated): AI recommends a specific action. A human confirms before execution occurs.

• T = 3 (Autonomous): AI executes without human intervention. This requires the highest level of implementation maturity and organizational trust.

Axis 2: Implementation and Maintenance Complexity (C)

This axis evaluates the technical friction in building and sustaining the automation, relative to the skills and resources of the team responsible for it. This is deliberately team-dependent. An automation rated C = 3 for a junior team may be C = 2 for a team of specialized AI engineers with established CI/CD pipelines for their playbooks.

• C = 1 (Low): Simple API calls or native integrations with minimal configuration.

• C = 2 (Medium): Multi-step orchestration across multiple systems requiring coordination and testing.

• C = 3 (High): Complex behavioral baselining, legacy system integration, or custom model tuning.

Axis 3: Operational Impact and Blast Radius (I)

This axis captures the business risk associated with the action. It is typically the most stable axis across organizations, but shifts based on asset criticality. Isolating a standard employee laptop has a different blast radius than isolating a production database server.

• I = 1 (Low): Negligible disruption. Background scans, tagging, enrichment activities.

• I = 2 (Medium): Temporary disruption. Resetting a standard user session, blocking a non-critical port.

• I = 3 (High): Significant downtime, data loss risk, or reputational damage. Production system changes, VIP account modifications, critical infrastructure alterations.

3.3 The Maturity Computation Logic

The scoring system builds from individual actions up to a full program assessment through five layers. Each layer uses a defined formula.

Layer 1: Action-Level Score (S)

For a single response action, the score is the sum of its three axis values:

The minimum possible score is 3 (T=1, C=1, I=1). The maximum is 9 (T=3, C=3, I=3).

Layer 2: Tier Mapping

The action score maps to one of four maturity tiers:

Score Range	Tier	Description
3.00 to 5.99	Explorer	Foundational; low-risk quick wins with minimal blast radius
6.00 to 6.99	Entry	Stabilized; moderate effort and impact, suitable for early-stage programs
7.00 to 7.99	Advanced	Mature; requires high-fidelity reasoning and established trust
8.00 to 9.00	Expert	Critical; high blast radius, autonomous VIP handling, or production-critical actions

Layer 3: Domain Maturity Score (D)

The maturity score for a specific domain (e.g., Endpoint, Identity) is the arithmetic mean of all action scores within that domain:

Where n is the number of scored actions in the domain. The resulting D value maps to a tier using the same thresholds from Layer 2.

Layer 4: Program Maturity Score (P)

The overall program score is the arithmetic mean of all domain scores, with equal weighting across all six planes:

Equal plane weighting is a deliberate design choice. It prevents planes with more actions (Endpoint has 22, SaaS has 10) from dominating the evaluation. Each plane contributes exactly one-sixth of the overall score.

Layer 5: Composite Maturity Label

The composite label is not derived from the program score directly. It uses sequential gating logic:

The composite label equals the highest tier where at least four out of six planes independently meet that tier's threshold, and the qualification chain is unbroken from Explorer upward. A product cannot be labeled Advanced if it has gaps at the Explorer tier.

The four-out-of-six rule is intentionally forgiving. A product focused on cloud-native environments may legitimately deprioritize network-level response. That should not disqualify it from a meaningful composite label. But it still needs breadth across most planes to earn a higher tier.

3.4 Context-Aware Scoring: Why Environment Matters

The ARMM recognizes that the maturity level of an automated action is not a static property of the feature itself. It is an emergent property of the environment where it is applied. The three axes (T, C, I) are all subject to organizational variance, which means the same product capability produces different scores in different contexts.

Example: "Isolate Device" evaluated by three different organizations using the same AI SOC product:

Context	Trust (T)	Complexity (C)	Impact (I)	Score	Tier
Org A: Mature Program / Expert Team	3	2	3	8	Expert
Org B: New Program / Junior Team	1	3	3	7	Advanced
Org C: High-Risk Assets / Manual-First	2	2	3	7	Advanced

The product capability is identical across all three. The scores differ because the Trust axis reflects implementation maturity, the Complexity axis reflects team capability, and the Impact axis (while stable here) can shift based on asset criticality. A vendor benchmark alone is insufficient. Builder Mode exists specifically to capture this variance.

4. Response Capability Domains

The framework organizes response capabilities into six domains. The first five (Identity, Network, Endpoint, Cloud, SaaS) cover specific technical response planes. The sixth (General Options / Usability) covers platform-level characteristics that affect the operational quality of the solution independent of any specific response action.

For the first five domains, each capability is scored using the 0-1-2 system described in Section 3.1 (Evaluator Mode) or the T+C+I system described in Section 3.3 (Builder Mode). For the General Options domain, the scoring criteria shift slightly: 0 means the feature is not available, 1 means the feature is available but limited in capability or partially implemented, and 2 means the feature is fully available, functional, and tested.

4.1 Identity Response Plane

Identity-related response actions target user accounts, service principals, groups, and access permissions. These actions are among the most commonly needed in incident response and are often the first automation candidates for SOC teams.

Action	Description
Reset Password	Reset a standard user's password
Revoke Sessions	Terminate all active sessions for a user account
Disable User	Disable a standard user account
Disable Service Principals	Disable a service account, service principal, or managed identity
Remove Permissions	Remove a specific set of permissions from an account
Group Adherence	Add or remove an account from a security group
Group Creation	Create a new security group
Token Rotation	Create or rotate secrets and tokens
Delete Sharing Permissions	Remove sharing permissions on resources
Label User (Tagging)	Apply a tag or label to a user account for tracking

Builder Mode Reference Scoring (Mature AI SOC Program, Skilled Engineering Team):

Action	T	C	I	Score	Tier
Group Adherence	3	1	1	5	Explorer
Label User (Tagging)	3	1	1	5	Explorer
Revoke Sessions	3	1	1	5	Explorer
Reset Password (Std)	3	1	2	6	Entry
Disable Standard User	3	1	2	6	Entry
Delete Sharing Permissions	2	2	2	6	Entry
Remove Specific Permissions	2	2	3	7	Advanced
Group Creation	3	2	2	7	Advanced
Disable Service Principals	2	2	3	7	Advanced
Reset VIP Password	3	2	3	8	Expert
Rotate Secrets (Prod)	2	3	3	8	Expert

4.2 Network Response Plane

Network-level response actions modify traffic flow, access control, and device connectivity. These are often high-impact actions with significant blast radius, making the Trust and Impact axes particularly important in scoring.

Action	Description
ACL Creation	Create a new access control list on the network
VLAN Creation	Create a new VLAN on the network
Firewall Rule Creation	Create a new firewall rule
IPS Rule Creation	Create a new IPS rule in deny mode
Network Connection Reset	Reset a network connection
DNS Entry Change	Modify an entry in the DNS records
Routing Table Change	Modify a routing entry
Sinkhole Traffic	Redirect traffic to a sinkhole
Rate Limit Traffic	Limit traffic by a particular indicator
VLAN Modification	Move a device to a restricted VLAN
Quarantine Device	Quarantine a device at the network level
Quarantine Server	Quarantine a server running an enterprise-level service
Modify NAT Rules	Change NAT rules to modify traffic patterns

Builder Mode Reference Scoring:

Action	T	C	I	Score	Tier
Network Connection Reset	3	1	1	5	Explorer
Sinkhole Traffic	3	1	1	5	Explorer
Rate Limit Traffic	3	1	1	5	Explorer
ACL Creation	2	2	2	6	Entry
Quarantine Device	3	1	2	6	Entry
Firewall Rule Creation	2	1	3	6	Entry
DNS Entry Change	2	2	2	6	Entry
Modify NAT Rules	2	2	2	6	Entry
IPS Rule Creation	2	2	3	7	Advanced
VLAN Creation	3	2	2	7	Advanced
VLAN Modification	3	2	3	8	Expert
Routing Table Change	3	3	3	9	Expert
Quarantine Server	2	3	3	8	Expert

4.3 Endpoint Response Plane

Endpoint response actions operate directly on devices and their software environment. This domain has the largest number of capabilities because endpoint response spans file operations, process management, application control, forensics, and OS-level changes.

Action	Description
Isolate Device	Isolate a device from all network connectivity
Initiate Malware Scan	Start a scan on the device
Grab File from Device	Upload a file to a designated container
Submit File to Sandbox	Submit a file for sandbox analysis
Lock Out User	Lock a user out of the device
Remove User from Device	Remove a user account from the device
Delete Files	Delete specific files from the device
Kill Processes	Terminate a running process
Remove Application	Uninstall an application
Remove Browser Extension	Remove a browser extension
Modify Browser Settings	Set, modify, or replace browser security parameters
Remove Scheduled Task	Remove a cron entry or scheduled task
Remove Startup Items	Remove a process, agent, or file from system startup
Remove Library / Package	Remove a library from a development environment
Upgrade Application	Force an automatic update on installed software
Upgrade OS	Force an automatic OS update
Deploy Script	Deploy a script or application needed for remediation
Modify Registry Key	Change a value or create a new registry key
Disable Service	Change the status of or remove a service
Collect Memory Dump	Initiate and retrieve a memory dump forensically
Clear Browser Cache	Remove all files, cookies, and data from the browser cache
Remove Device from Domain	Remove a device from the domain

Builder Mode Reference Scoring:

Action	T	C	I	Score	Tier
Initiate Malware Scan	3	1	1	5	Explorer
Clear Browser Cache	3	1	1	5	Explorer
Grab File from Device	3	1	1	5	Explorer
Collect Memory Dump	2	2	1	5	Explorer
Submit File to Sandbox	3	1	1	5	Explorer
Kill Processes	3	1	2	6	Entry
Block File (via Hash)	3	1	2	6	Entry
Lock Out User	2	2	2	6	Entry
Remove Browser Extension	3	1	2	6	Entry
Remove Scheduled Task	2	2	2	6	Entry
Remove Startup Items	2	2	2	6	Entry
Disable Service	2	2	2	6	Entry
Delete Files	2	2	2	6	Entry
Modify Browser Settings	2	2	2	6	Entry
Remove Application	2	2	3	7	Advanced
Remove User from Device	2	2	3	7	Advanced
Remove Library / Package	2	2	3	7	Advanced
Modify Registry Key	2	3	3	8	Expert
Isolate Device	3	2	3	8	Expert
Remove Device from Domain	2	3	3	8	Expert
Upgrade Application	2	3	3	8	Expert
Upgrade OS	2	3	3	8	Expert
Deploy Script	3	3	3	9	Expert

4.4 Cloud Response Plane

Cloud response actions target infrastructure resources, access controls, and storage in cloud environments. The blast radius of cloud actions can be particularly severe because a single misconfigured change can affect multiple dependent services.

Action	Description
Modify Security Group Rules	Modify firewall rules on a cloud resource to restrict access
Create Security Group	Create a new security group and apply it to restrict traffic
Isolate Resource	Quarantine a cloud resource so it is unreachable
Modify Access Type	Switch a resource from public to private or restrict anonymous access
Remove Permissions to Resource	Remove a service principal or managed identity from accessing a resource
Delete Resource	Delete a resource from the cloud environment
Stop Resource	Stop a resource from execution
Modify KeyVault Entries	Add or modify resources in a KeyVault
Use Breakglass Account	Use a breakglass account in case of emergency
Remove Files from Storage	Remove files from a storage bucket or storage account
Copy Storage Device	Create a copy of a cloud storage resource for forensic investigation
Mount Storage Device	Mount a new storage capability to a VM for forensic investigation
Snapshot VM	Create a snapshot of the current state of a virtual machine
Enable Diagnostic Settings	Alter settings that enable advanced log gathering
Apply Resource Lock	Make the resource immutable or read-only

Builder Mode Reference Scoring:

Action	T	C	I	Score	Tier
Enable Diagnostic Settings	3	1	1	5	Explorer
Apply Resource Lock	3	1	1	5	Explorer
Snapshot VM	3	1	1	5	Explorer
Stop Resource	3	1	2	6	Entry
Modify Security Group Rules	2	2	2	6	Entry
Create Security Group	2	2	2	6	Entry
Remove Permissions to Resource	2	2	2	6	Entry
Copy Storage Device	2	2	2	6	Entry
Mount Storage Device	2	2	2	6	Entry
Modify Access Type	2	2	3	7	Advanced
Isolate Resource	3	2	3	8	Expert
Remove Files from Storage	2	2	3	7	Advanced
Modify KeyVault Entries	2	3	3	8	Expert
Use Breakglass Account	2	3	3	8	Expert
Delete Resource	2	3	3	8	Expert

4.5 SaaS Response Plane

SaaS response actions focus primarily on email and productivity platforms, which are among the most common attack surfaces in enterprise environments. Actions in this domain directly affect end-user workflows and communications.

Action	Description
Delete Email	Remove an email from a user's mailbox
Quarantine Email	Move an email to the user's quarantine or junk box
Create Routing Rules	Create rules to handle and route incoming email
Grab Email Sample	Extract an attached file from an email
Grab Email Link	Extract a link from inside an email message
Add / Remove Meeting Invite	Modify a user's calendar
Read / Modify User Status	Read or change a user's status in the HR platform
Disable Malicious Inbox Rule	Disable a rule created by a malicious actor from a user's mailbox
Block Sender	Block a sender from the domain
Modify HR Records	Modify HR records in the system beyond status

Builder Mode Reference Scoring:

Action	T	C	I	Score	Tier
Enable Diagnostic Settings	3	1	1	5	Explorer
Apply Resource Lock	3	1	1	5	Explorer
Snapshot VM	3	1	1	5	Explorer
Stop Resource	3	1	2	6	Entry
Modify Security Group Rules	2	2	2	6	Entry
Create Security Group	2	2	2	6	Entry
Remove Permissions to Resource	2	2	2	6	Entry
Copy Storage Device	2	2	2	6	Entry
Mount Storage Device	2	2	2	6	Entry
Modify Access Type	2	2	3	7	Advanced
Isolate Resource	3	2	3	8	Expert
Remove Files from Storage	2	2	3	7	Advanced
Modify KeyVault Entries	2	3	3	8	Expert
Use Breakglass Account	2	3	3	8	Expert
Delete Resource	2	3	3	8	Expert

4.6 General Options / Usability

This domain evaluates platform-level capabilities that are not tied to any specific response action but directly affect how useful, trustworthy, and manageable the AI SOC solution is in production. The scoring for this domain uses a modified scale: 0 means not available, 1 means available but limited, and 2 means fully available and functional.

This domain is split into two sub-categories to distinguish between operational platform features and AI-specific evaluation criteria.

Platform Operations

Action	Description
Close Alerts in SIEM	The tool can close alerts in all major SIEM solutions
Logging	Platform logging allows identification of all actions taken
Reasoning Logging	Reasoning steps taken by the platform are logged at sufficient detail
API Development	The API is robust enough for integration with other security tools
Support Level	Support is responsive and allows for adequate issue resolution
Account Management	Account management is straightforward with SSO integration
Roles and Responsibility	Role-based access control is available with sufficient granularity
Ease of Use (GUI)	The GUI is navigable and intuitive
Native Chat Integration	Native integration with major communication platforms
Alerting	Automatic alerting when platform-level or analysis-level issues arise
Stats / Health Dashboards	Dashboards showing current platform status and performance

AI-Specific Evaluation Criteria

Action	Description
Bring Your Own Model	Ability to integrate custom models into the platform
Context Grounding	Ability to bring organizational data to feed into the ML model
Autonomous Action Thresholds	Platform allows setting confidence thresholds for autonomous execution
Investigation Audit Trail	Complete, exportable record of every action (AI and human) with timestamps
IR Metrics Tracking	Native tracking of MTTD, MTTA, MTTT, MTTI, MTTR without external tooling
Feedback Loop Mechanism	Analysts can confirm, reject, or correct AI decisions with feedback incorporated
Auto-Close Reversal Tracking	Tracks the rate at which auto-closed alerts are reopened by analysts
Explainability / Decision Transparency	AI provides clear, traceable reasoning for every decision
AI Decision Accuracy Reporting	Tracks TP accuracy, FP accuracy, and confidence scores over time
Model Drift Detection	Monitors AI model performance and alerts when accuracy degrades
Adversarial Robustness Testing	Supports or integrates with red team exercises to test AI resilience

5. Aggregate Maturity Scoring

Evaluating each plane individually is necessary but not sufficient. Security teams making purchasing decisions and product managers tracking competitive positioning need a consolidated view that communicates the overall picture without hiding the details.

5.1 Automation Depth Score

This is the most operationally significant metric and the one that separates real autonomous solutions from products that wrapped a chatbot interface around a set of API calls.

Across all covered capabilities, calculate the distribution:

• What percentage is fully automated (level 2)?

• What percentage sits at Approver level (1A)?

• What percentage sits at Guide level (1G)?

• What percentage sits at Collaborator level (1C)?

• What percentage is not available at all (0)?

A product could have 80% of capabilities covered but only 5% fully automated. That is a fundamentally different product than one with 60% covered but 40% fully automated. The first is broad but shallow. The second is narrower but operates with real autonomy where it counts.

Full Automation Rate: The percentage of total capabilities at level 2. This is the true measure of how much an AI SOC solution can operate without human intervention.

Coverage Rate: The percentage of total capabilities at any level above 0. This measures breadth regardless of automation depth.

The relationship between these two numbers tells you everything about how the product actually operates. A high coverage rate with a low automation rate means the product is a guided workflow tool with AI branding. A moderate coverage rate with a high automation rate relative to coverage means the product is autonomous in its areas of focus but limited in scope.

5.2 Combined Scoring Readout

A complete ARMM evaluation for a product produces the following consolidated output:

Metric	Value
Overall Score	47% (equal plane weighted)
Composite Maturity	Entry (5 of 6 planes at Entry or above)
Automation Depth	12% fully automated, 61% covered at any level

Per-Plane Breakdown:

Plane	Score	Coverage	Fully Automated	Tier
Identity	78%	7/9 covered	2 actions	Advanced
Network	42%	8/13 covered	1 action	Entry
Endpoint	38%	12/21 covered	1 action	Entry
Cloud	31%	6/15 covered	0 actions	Explorer
SaaS	55%	7/10 covered	3 actions	Entry
General Options	64%	10/14 covered	3 actions	Advanced

6. Reading the Model

A product can reach Expert level on a specific plane by checking all the boxes for that domain. But it would be difficult to consider an AI SOC Response solution as Expert level overall if it lacks the ability to perform foundational actions like closing alerts in a SIEM. The tier system is designed to reward both depth within a domain and breadth across domains.

The reference maturity tables provided in Section 4 use example scores from a hypothetical mature AI SOC program with a skilled engineering team. These are illustrative, not universal benchmarks. The environmental dynamics described in Section 3.5 are not optional context; they are a core part of how the framework is intended to be used.

When comparing two products, the most informative comparison is not the aggregate score. It is the per-plane breakdown combined with the Automation Depth Score. Two products at the same composite tier can have radically different operational profiles. One may cover 80% of capabilities at the Collaborator level. The other may cover 50% but with 30% at full automation. These are different products for different buyers with different operational maturity levels.

7. Limitations and Future Work

This is version 0.1. The framework has known limitations:

• The capability lists are not exhaustive. New response actions will emerge as AI SOC products mature and as attack surfaces expand.

• The three-axis scoring (T, C, I) requires subjective judgment that will vary between evaluators. We plan to develop calibration guidelines to reduce inter-evaluator variance.

• The framework does not currently weight domains differently. In practice, Identity response may be more important than Network response for a given organization. Weighted scoring is planned for a future version.

• Detection and analysis capabilities are out of scope for this version. A separate framework or an extension to ARMM may address those in the future.

• We have not included pricing, deployment time, or vendor lock-in considerations. These are important purchase factors but are outside the scope of a technical maturity model.

We are building a public web application where users can input their product's capabilities and generate ARMM scoring layers automatically, along with an exportable CSV. The application is available at: armm.secops-unpacked.ai

8. Conclusion

The AI SOC market is growing faster than the industry's ability to evaluate products on consistent terms. The ARMM framework provides a structured, repeatable methodology for measuring what an AI SOC solution can actually do in the response layer, how autonomously it can do it, and what it takes to deploy and maintain that capability in a specific operational environment.

The framework is built for two audiences: security teams evaluating products and product managers building them. For security teams, it provides a checklist and scoring system that cuts through marketing language and focuses on operational capability. For product teams, it provides a competitive analysis baseline and a prioritization framework for feature development.

SOAR gave us arms without brains. The first wave of AI SOC products gave us brains without arms. The products that will win this market are the ones that connect both. ARMM gives you a way to measure how far along that connection is, and where the gaps remain.

No current AI SOC solution will check every box. That is not the point. The point is to establish a common language and a common measurement system so that the conversation about AI SOC response capability is grounded in specifics rather than promises. Version 0.1 is the starting point. The framework will evolve as the market does.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

The Fear of Not Doing Enough: Security's Workflow Problem

If you've been following this blog, you know I've spent a lot of time on AI transforming investigation, triage, and detection engineering. And a few months back I wrote about the single pane of glass, how it's not a product you buy but a system you build, piece by piece, like Legos.

That post was about the architecture. What tools do you need, how do they connect, where does data flow.

This post is about the layer underneath that nobody talks about. Not the tools. The work itself. Where it comes from, how it flows, and why we have zero visibility into most of it.

The Fear of Not Doing Enough

Security has this pattern I keep seeing everywhere.

New attack technique drops. A CVE trends on Twitter. Some threat intel report lands in your inbox with a fancy APT name. What happens next? Predictable.

Someone writes a generic detection rule fast so the team has "something." Gets pushed to production. Generates noise. Nobody tunes it because there's already another thing screaming for attention.

The false sense of coverage becomes more important than actual coverage.

I call this the Fear of Not Doing Enough. And honestly? It drives most of the operational pain in security teams today.

You write a detection rule. Do you have the SOP for when it fires? Do you know the full analysis path an analyst should follow? Can you estimate how that alert impacts your team's workload downstream? Do you know what "done" looks like for that alert type?

If you can't answer those, you didn't deploy a detection. You deployed a work generator with no operating manual. Multiply that across dozens of detections written under pressure and you get patchwork coverage that looks great on a dashboard but falls apart when someone has to actually operate it.

But here's the thing. Even if you fix all of that, you're still only looking at one input stream.

It's Not Just SIEM Alerts

A ACM Computing Surveys paper (Tariq et al., 2025) reviewed over 30 solutions to alert fatigue in SOCs. Thorough paper, I'll give them that. Identifies four root causes: staff shortage, high false positive rates, disconnected dashboards, and inefficient SOPs.

But every single solution assumes the work starts with a SIEM alert.

Now look, I'm not saying SIEM alerts are a small part of the work. For most teams they're probably more than half. But here's what matters: the work that doesn't come from the SIEM is often the most manual, least structured, and hardest to track.

IT escalations. Someone from the help desk pings you on Slack: "Hey, this looks weird." Access review requests from HR. Audit findings that need remediation tracking. Pen test findings that need to be assigned and fixed. Third-party risk questionnaires. Compliance asks from legal.

All real security work. And here's the thing about it: only some or none of it has a playbook or automation behind it. Most of it lives in Slack threads, email chains, and spreadsheets. It's the security work that runs entirely on copy-paste, tribal knowledge, and good intentions.

Your SIEM alerts, for all their problems, at least flow through a pipeline. They get enriched. They have some structure. Maybe even a SOAR playbook attached. The non-SIEM work? It's the Wild West.

Erik Bloch has been making this point for years.A lot of the work SOC is doing day-to-day has nothing to do with chasing advanced adversaries. It's tickets, reports, evidence collection, reconciling data across tools. The mundane operational grind that actually burns people out.

And here's the part that really gets me. Outside of very large enterprises that have 10 security sub-departments with dedicated teams for everything, the same 3-5 people triaging SIEM alerts are also pulling evidence for the auditor, handling the IT escalation, and answering the compliance questionnaire. There's no luxury of specialization. The alert queue is just one input stream among many. And the non-SIEM stuff eats time disproportionately because it's all manual.

Security Work Has No Gravity

Ross Haleliuk recently wrote a great piece about ServiceNow betting on "workflow gravity" to compete with the security platform giants. The thesis is simple. Whoever owns where work happens owns the decisions.

Data gravity pulls information into a single system of record. Your SIEM, your data lake, whatever. That part most teams have figured out. Workflow gravity is different. It pulls action into a single system of action. One place where work lands, gets triaged, gets tracked, and gets done.

Right now? Security work has no gravity. It's everywhere and nowhere.

And yeah, this connects directly to the single pane of glass conversation. In that post I talked about building your own platform, Lego-style, with assets, data layers, correlation, and response actions. But even if you build that beautiful architecture, it's still oriented around machine-generated alerts. The SIEM brain, the enrichment layer, the correlation engine. All of that assumes the input is a structured alert.

What about the IT manager who emails you about a suspicious contractor? What about the audit finding that needs 6 teams to remediate? What about the pen test report sitting in a shared drive that nobody has turned into action items yet?

That work has no architecture. It has no pipeline. It just shows up and someone deals with it however they can.

You want to know why security teams always feel understaffed? Part of it is real headcount shortage, sure. But part of it is that nobody can actually see where the time goes. When the most manual, time-consuming work lives outside of every system you've built, you can't measure it. When you can't measure it, you can't optimize it. When you can't optimize it, you just throw more people at it and hope for the best.

Process Mining Exists. Just Not for Us. Yet.

Here's something that gets me. In finance, procurement, and operations, tools like Celonis and Scribe Optimize have existed for years. They observe how work actually happens across tools and systems. They find bottlenecks. They tell you where time is wasted. They optimize based on data, not vibes and assumptions.

In security? Still very early days.

Some vendors are starting to take RPA-style approaches to There's a handful of academic papers exploring it. But it's nowhere near mainstream.

We still don't have good data on how security work actually flows end to end. Think about that.

We have terabytes of security telemetry. We can tell you exactly when a process spawned on an endpoint at 3:47am. But we can't tell you how long it takes an analyst to go from "alert fired" to "investigation complete." We can't tell you how much time the team spends on compliance requests versus actual threat work. We can't tell you which of your 200 detection rules generates the most operational overhead relative to the security value it provides.

That's wild.

Why This Is Hard

I get why the industry keeps gravitating toward the easier wins. Make investigation faster. Automate the playbook. Build a better ML model for triage. Those are well-defined problems with measurable outcomes.

Understanding where all security work happens and how it flows? That's messy. It crosses tool boundaries. It involves human behavior that doesn't fit neatly into event logs. It requires looking at the whole system, not just one piece.

This is the hardest problem to solve. And that's exactly why not many are tackling it yet.

But here's why it matters. If you don't understand the full picture of how work enters and flows through your security team, everything else you build is an optimization of a subsystem. You can make SIEM triage 10x faster, but if a third of the work comes from non-SIEM sources that are entirely manual, you just made one part of the problem better while the messiest part stays untouched.

What Would Actually Help

I don't think this needs to be one giant platform that replaces everything. But teams need a few things that barely exist today.

Workflow data. How long does each type of work actually take? Where are the handoffs? Where do things stall? What percentage of the team's time goes to which category of work? Right now most teams are guessing. And the guesses are usually wrong because the most painful work is the least visible.

Operational impact awareness. Before you deploy a new detection, onboard a new data source, or agree to a new compliance requirement, you should be able to model what that does to your team's capacity. Not after the fact when everyone's drowning. Before.

Connection between detection and process. If you have a detection but you don't have the analysis path mapped from it, you can't estimate how it impacts anything downstream. Every detection should ship with its SOP. Not as a nice-to-have. As a requirement.

The Fear Won't Go Away

The Fear of Not Doing Enough will always be there. New threats aren't going to stop coming. The pressure to have "something" for every new attack vector is real.

But the answer isn't to keep throwing generic detections at every new thing and hoping the team can absorb the blast. It's not to keep building faster investigation tools for one slice of the work while the rest drowns in Slack threads and spreadsheets.

We've been fixing the middle. Investigation is getting faster. AI triage is real. Response automation is improving. The single pane of glass architecture is getting clearer. All good progress.

Now it's time to zoom out. Understand how security work actually flows. All of it. Not just the structured, machine-generated part. Especially the messy, manual, human-generated part that eats the most time and has the least tooling.

Fix the input. Model the cost. Understand the workflow.

Stop optimizing the output of a system you've never fully mapped.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Hey everyone. Been quiet on here for a bit. First post of 2026.

I came across a paper that finally articulates something I've been thinking about for a while: autonomy isn't a capability score. It's a design decision"Levels of Autonomy for AI Agents". And chasing the highest level everywhere is a mistake.

Let me walk through it.

L1 (Operator): User directs and makes decisions, agent acts. Think Microsoft Copilot. The agent requires invocation to act, provides on-demand assistance, and avoids preference-based decision-making on the user's behalf.

L2 (Collaborator): User and agent collaboratively plan, delegate, and execute. Think OpenAI Operator. Users can freely modify agent work and take control at any point. Back-and-forth communication is frequent.

L3 (Consultant): Agent takes the lead but consults user for expertise and preferences. Think Gemini Deep Research. Users provide feedback and directional guidance rather than hands-on collaboration. The agent bears more of the learning curve.

L4 (Approver): Agent engages user only in risky or pre-specified scenarios. Think Devin. Users specify approval conditions upfront. The agent only stops for blockers, credentials, or consequential actions.

L5 (Observer): Agent operates with full autonomy under user monitoring. Users can watch activity logs and hit the emergency stop. That's it.

The key insight: autonomy is a design decision, not a capability metric. A capable agent can still operate at L2 if that's the right call for the task. The paper explicitly argues against treating autonomy as an inevitable consequence of increasing capability.

Agency vs. Autonomy

The paper makes an important distinction that matters for security operations.

Agency is the capacity to carry out intentional actions. It's about what tools the agent has access to and what it can do in the environment.

Autonomy is the extent to which the agent operates without user involvement. It's about when and how the agent checks in with humans.

An agent with high agency (many tools, broad permissions) can still have low autonomy (checks in frequently). An agent with low agency (limited toolset) can have high autonomy (runs independently within that scope).

This distinction matters because security teams often conflate the two. Giving an agent access to more data sources (agency) is different from letting it act without approval (autonomy). You can expand agency while constraining autonomy.

Mapping Autonomy to Security Operations

I mapped common security workflows to appropriate autonomy levels based on their risk profile and decision complexity.

Initial Triage: L4-L5

For alert triage, scale beats precision. You're dealing with volume. The goal is filtering, not final judgment.

L4 makes sense here. Let agents do the heavy lifting, have them seek approval only for edge cases or high-severity alerts. L5 is reasonable for low-fidelity alerts where false positives cost nothing. Start at L4. Keep humans reviewing outcomes for anything that escalates to investigation.

The paper notes that L4 agents are ideal for tasks with high amounts of lower-stakes decision-making. Alert triage fits this description. Automated decisions improve efficiency. Erroneous decisions on individual alerts don't impose catastrophic risks if the escalation path is intact.

Incident Response: L1-L2

On the response side of the IR cycle, L1 and L2 work well. These are established patterns with runbooks and playbooks.

Why keep autonomy low here? Response actions are consequential. Isolating a host, blocking a domain, killing a process. These actions have real impact. Speed matters, but accountability matters more.

L1 is appropriate when analysts drive the workflow and agents execute specific tasks on command. L2 works when you want the agent to propose containment plans while the analyst retains takeover capability.

The paper describes L2 control mechanisms as requiring "control transfer from agent to user, and vice versa" plus "shared representation of progress." That maps well to incident response dashboards where analysts can see what the agent is doing and intervene.

In-Depth Investigation: L3

Deep investigation is judgment-heavy work. Context matters. The analyst brings domain knowledge, institutional memory, and threat intelligence the agent doesn't have.

L3 fits this workflow. The agent leads the investigation, gathering data, correlating events, building timelines. But it consults the analyst for direction. What's the hypothesis? Which threads are worth pulling? Does this pattern match something we've seen before?

The paper notes that L3 agents require "productive and timely consultation." The agent needs to know what expertise the user brings and when to ask for it. For security investigations, this means the agent should surface findings and ask about relevance rather than drawing conclusions autonomously.

Threat Hunting: L2-L3

Hunting is exploratory by nature. You're looking for things you don't know exist yet. Hypotheses matter. Intuition matters.

Collaboration beats full automation here. L2-L3 is the range. The agent surfaces anomalies, suggests investigation paths, runs queries. The human drives the hunt itself.

The paper describes L2 as the level where "back-and-forth communication between the user and the agent is the most frequent and rich." Threat hunting benefits from this dynamic. The hunter's domain expertise combined with the agent's ability to process large datasets creates a feedback loop that pure automation can't replicate.

Detection Engineering: L2-L3

Detection engineering is systematic but consequential. Bad detections create alert fatigue. Missed detections create gaps.

L2 is the baseline. The agent assists with query building, suggests detection patterns, helps test against historical data. The engineer retains control over what gets deployed.

L3 is appropriate for mature teams with well-governed detection lifecycles. The agent drafts detections, runs validation, and consults the engineer before deployment. The key is having proper testing and review controls already in place.

The paper warns about L4 agents and "meaningless rubber stamping" from user disengagement. This risk is real for detection engineering. If engineers just approve whatever the agent proposes, detection quality will degrade.

The Double-Edged Sword

The paper repeatedly emphasizes that autonomy amplifies both benefits and risks. Higher autonomy means more scale and efficiency. It also means errors compound over multiple steps without intervention.

This maps directly to security operations. An agent that autonomously closes false positive alerts at L5 saves analyst time. An agent that autonomously closes true positive alerts at L5 creates security incidents.

The paper also raises concerns about deskilling and loss of critical thinking when automation takes over judgment tasks. Security teams should consider this. If agents handle all investigation, what happens to analyst skill development? L2 and L3 autonomy levels preserve opportunities for human engagement while still providing automation benefits.

Autonomy Certificates

The paper proposes "autonomy certificates" as a governance mechanism. A third-party body evaluates an agent's behavior and certifies the maximum autonomy level at which it can operate.

This concept has implications for security vendors. Right now, every AI SOC vendor claims some version of autonomous operation. There's no standard way to compare what that actually means.

An autonomy certificate framework would force clarity. Does your agent operate at L3 or L4? What approval mechanisms exist? Under what conditions does it escalate?

For security buyers, this creates better evaluation criteria than vague claims about AI capabilities.

Double-Layer Governance: Reasoning and Abilities

The agency vs. autonomy distinction from the paper points to a practical governance model. You need to control both what the agent can think about doing and what it can actually do.

At BlinkOps, we implement this as double-layer governance:

Layer 1: Reasoning Constraints

This layer limits what the agent can decide to do. It's autonomy governance. You define the scope of problems the agent is allowed to reason about and the types of conclusions it can reach.

For example, an agent handling alert triage might be constrained to reason only about severity classification and enrichment. It can't decide to initiate response actions, even if it has the technical capability. The reasoning boundary is set before the agent ever considers what actions to take.

This maps to the paper's definition of autonomy as "the extent to which an AI agent is designed to operate without user involvement." By constraining reasoning scope, you limit how far the agent goes before involving a human.

Layer 2: Ability Constraints

This layer limits what the agent can execute. It's agency governance. Even if the agent reasons its way to a valid conclusion, it can only act through explicitly permitted capabilities.

This is your tool allowlist. The agent might determine that isolating a host is the right response, but if host isolation isn't in its permitted action set, it can't execute. It has to escalate.

This maps to the paper's definition of agency as "the capacity to carry out intentional actions." By constraining the toolset, you bound the blast radius of any autonomous decision.

Why Both Layers Matter

Single-layer governance creates gaps.

If you only constrain abilities (Layer 2), the agent can still reason about actions outside its scope and make recommendations that push humans toward decisions the agent shouldn't influence. An agent without response permissions might still conclude "this host should be isolated immediately" and create pressure for hasty action.

If you only constrain reasoning (Layer 1), the agent might find edge cases where its reasoning scope overlaps with dangerous capabilities. A triage agent reasoning about "enrichment" might decide that querying a production database for context falls within scope.

Double-layer governance closes both gaps. The reasoning layer defines intent boundaries. The ability layer enforces execution boundaries. An action only happens if it passes both checks.

Practical Implementation

For each workflow, define:

1. Reasoning scope: What questions can the agent answer? What conclusions can it reach? What types of decisions are out of bounds?

2. Action permissions: What tools and integrations can the agent invoke? What parameters can it set? What requires human approval?

3. Escalation triggers: When reasoning hits scope boundaries, where does it go? When actions require approval, who approves?

This gives you granular control without blocking automation entirely. An L4 agent can still operate autonomously within its defined scope. But that scope is explicitly bounded at both the reasoning and execution layers.

The paper's framework helps here. L4 requires "customizable conditions for seeking approval." Double-layer governance operationalizes this. The conditions are defined by reasoning scope violations (Layer 1) and action permission requirements (Layer 2).

What This Means for AI SOC Design

If you're building or buying AI-powered security tooling, ask different questions:

What autonomy level does this workflow need? Not "how autonomous is this agent?" Match the autonomy to the task risk profile.

What are the must-have controls? Each autonomy level has required control mechanisms. L4 requires approval elicitation for consequential actions and customizable conditions. L2 requires control transfer mechanisms and shared progress visibility. Verify these exist.

Where are the approval gates? Every workflow should have defined checkpoints. Know what triggers human involvement.

What's the fallback? When the agent hits a failure state, what happens? The paper notes that L4 and L5 agents should iterate on solutions or modify approaches when blocked. How does your agent handle this?

Who's accountable? Higher autonomy means harder accountability tracing. The paper cites research showing it's simultaneously more important and more difficult to anticipate harms from autonomous AI. Design governance around this reality.

Closing Thoughts

Chasing L5 everywhere is a design mistake, not a strategy.

The vendors pushing "fully autonomous SOC" are selling a destination most teams shouldn't want to reach. The right autonomy level varies by task, by maturity, by risk tolerance.

The paper's framework gives us a shared vocabulary for these discussions. Use it.

Reference: Feng, K.J.K., McDonald, D.W., & Zhang, A.X. (2025). Levels of Autonomy for AI Agents. University of Washington. arXiv:2506.12469

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Article written in collaboration with Andrew Green

We often argue over semantics, but your agents shouldn't. They need explicit definitions for what data and events mean for your business. Otherwise they will refer back to their training data, which is often not applicable.

For example, a log from an on-prem Cisco router and an AWS VPC log are structured differently, but they both contain IP addresses, ports, protocols. An LLM can understand what these network elements mean and make general inferences about network requests. But how will it determine which instances of traffic between your on-prem environment and cloud are expected versus suspicious?

You've got three options here.

The first one is to ask the LLM: 'Hey, is this suspicious?', at which point the LLM will be using its training data and prompt context to give a statistically likely interpretation. This is the neural approach.

The second one is to define an explicit rule, such as alert if request.source_ip in "10.0.0.0/8" and request.destination_ip in "52.0.0.0/8". This is the symbolic approach.

The third one is to tell the LLM: 'Hey, our dev Jerry had a leg surgery and he won't leave the house while he heals, but he'll be working from home'. With the right tools, the LLM will now interpret the ambiguities of this message and combine them with hard rules to define an explicit instruction, where all of Jerry's access requests will be made from his VPN for the next 6 months, while anything else is suspicious. This is the neurosymbolic approach.

To get to this third option, we need a semantic layer.

The Fundamental Tension: Why We Need Semantic Layers

Before we define what semantic layers are, let's understand the problem they're solving. There's a fundamental tension in how we approach automated reasoning in security:

Pure Neural (LLM) Approach:

• Excellent at pattern recognition and handling ambiguity

• Can process natural language queries

• Adaptable to new situations

• BUT: Non-deterministic, expensive at scale, can't explain decisions

Pure Symbolic Approach:

• Deterministic and auditable

• Fast and efficient at scale

• Provides clear reasoning chains

• BUT: Brittle with edge cases, requires extensive rule maintenance

The semantic layer provides a foundation for hybrid approaches where symbolic reasoning handles the facts and constraints while neural networks handle the ambiguity and interface. It's the bridge between "Jerry is working from home while recovering" (natural language) and "source_ip = Jerry_VPN_IP AND time_range = next_6_months" (executable logic).

But What Is a Semantic Layer, Exactly?

A semantic layer is an LLM-queriable abstraction layer that pulls and correlates information about entities, their relationships, and the deterministic rules that govern their interactions. 

It correlates raw technical data with business meaning through symbolic reasoning.

But what is symbolic reasoning?

Actually, what is a symbol?

A symbol is a notation that captures the meaning of a concept, such as 'putting two things together' is expressed by the symbol '+'. Humans and LLMs can use these symbols to read and define rules and instructions.

Symbolic reasoning therefore uses these notations to work out a conclusion.

At its core, a semantic layer must define the following three concepts:

1. Entities: identities (both human and workload), assets such as servers and applications, events like a GET request, and behaviors such as Dev (identity) makes a POST request (event) to a database (asset).
   

2. Relationships also known as relationship graphs or knowledge graphs, these map things like User X manages Asset Y, Asset Y hosts Application Z, Application Z processes Data Type W, Data Type W is subject to Regulation V.
   

3. Symbolic Reasoning: applying logic-based rules. You can trace exactly why a decision was made. Every step is auditable, testable, and deterministic.
   
   a. IF user.role = "contractor"
   
   b. AND access.time NOT IN business_hours
   
   c. AND asset.classification = "confidential"
   
   d. THEN risk_score = HIGH
   

Where Should Semantic Layers Live?

Considering most security activities revolve around gathering accurate and relevant data, where in the modern SecOps stack should this semantic layer be inserted?This is where theory meets painful reality, and where practitioners diverge sharply on the right approach.

There are four main architectural positions, each with compelling arguments and serious tradeoffs:

Option 1: In the Data Pipeline (Pre-SIEM)

The argument here is to do semantic processing before data ever reaches your SIEM or data lake.

Advantages:

• Reduces data volume through intelligent filtering

• Normalizes entities at the source

• Cheaper than processing in expensive SIEM platforms

• Can route data based on semantic understanding

Challenges:

• Requires real-time processing at massive scale

• Changes to semantic models require pipeline updates

• Limited context (can't look at historical data easily)

• Becomes another system to maintain

Verdict: This works well for basic entity resolution and enrichment, but struggles with complex relationship modeling that requires historical context.

Option 2: Within the SIEM Platform

Modern SIEMs increasingly claim to embed semantic capabilities directly.

Advantages:

• Integrated with detection and investigation workflows

• Access to all historical data for context

• Single platform to manage

• Vendor support (in theory)

Challenges:

• Vendor lock-in to proprietary semantic models

• Performance impacts on already-stressed SIEMs

• Limited flexibility in modeling

• Expensive processing at SIEM rates

Verdict: most SIEM vendors are adding "semantic" features, but they're often just rebranded lookup tables and data models. True semantic reasoning with relationship graphs and symbolic logic remains limited. XDR vendors have marketed this capability more aggressively, positioning themselves as solving the "semantic gap" problem, but implementation depth varies significantly.

Option 3: As a Separate Analytics Layer

Building a semantic layer that operates alongside or on top of your security data infrastructure.

Advantages:

• Flexibility to model your specific environment

• Can work across multiple data sources

• Optimized for complex reasoning

• Natural fit for AI/ML integration

Challenges:

• Another platform to integrate and maintain

• Potential latency for real-time use cases

• Requires data federation or complex APIs

• Organizational boundaries (who owns it?)

Verdict: This is where most successful implementations end up, but it requires significant investment and organizational commitment. Some SOAR platforms have evolved toward this model, building semantic layers with automation workflows and embedding business context in case management. AI SOC platforms are also exploring this space, though many are still in early stages of true semantic reasoning versus simple enrichment.

Option 4: Distributed Semantic Processing

The most ambitious approach - semantic capabilities distributed throughout your stack, with processing happening wherever it makes most sense.

Advantages:

• Process data where it makes most sense

• No single point of failure

• Can optimize for different use cases

• Scales naturally with infrastructure

Challenges:

• Consistency across distributed models

• Complex orchestration and governance

• Difficult to maintain and update

• Requires sophisticated engineering

Verdict: This is the "microservices of semantic layers" - sounds great in theory, nightmarish in practice for most organizations. You need mature DevOps practices and significant engineering resources to make this work.

How Semantic Layer Projects Fail

Let's be honest about the failure modes, because understanding these is more valuable than any architecture diagram:

1. Trying to Model Everything

Organizations attempt to create comprehensive ontologies of their entire environment. This is impossibly complex and never finishes. You don't need a complete knowledge graph of your infrastructure to get value from semantic layers. But you do not explicit information about what is modeled and what is not.

2. Ignoring Organizational Reality

Semantic models require agreement on basic concepts like "what is a critical asset?" Most organizations can't even agree on this informally, much less formally model it. The technical challenge is often easier than the political one.

3. Underestimating Maintenance

Semantic models aren't write-once. They require constant updates as your environment evolves. Without dedicated resources, they become stale and useless within months.

4. Over-Engineering the Solution

Building elaborate graph databases and reasoning engines when simple lookup tables would suffice for 80% of use cases. Start with the minimum viable semantic layer that solves your specific problems.

5. Lacking Clear Use Cases

Implementing semantic layers because they sound advanced, not because they solve specific problems. "We need better context" isn't a use case - "We need to automatically identify which database access is anomalous based on team structure and data classification" is.

The Practical Path Forward

If you're considering implementing semantic capabilities, here's what actually works:

Start Small and Specific

Pick one use case with clear business value. User-to-asset relationship modeling for access analysis. Application dependency mapping for incident scope. Data classification for DLP prioritization. Prove value before expanding.

Embrace Imperfect Models

Your semantic layer doesn't need to be complete or perfect. Even modeling 60% of your critical relationships provides massive value over having none. Accept that edge cases will exist.

Build for Maintenance

Plan for model updates from day one. Who owns entity definitions? How do relationships get updated? What's the approval process for changes? The technical implementation is secondary to the governance model.

Leverage Standards Where Possible

Don't reinvent entity definitions that already exist. Frameworks like OCSF provide common semantic models that reduce custom development. Think of these as starting points, not constraints.

Consider Buy vs Build Carefully

Building custom semantic layers requires significant engineering investment. Many organizations would be better served by vendor solutions, even with their limitations. Be honest about your capabilities and resources.

As well some great resources:
https://www.holistics.io/books/setup-analytics/data-modeling-layer-and-concepts/

https://www.datacamp.com/blog/semantic-layer

https://arxiv.org/html/2506.17512v1

https://arxiv.org/html/2510.16610v1

Speculation on The Future of Semantic Layers in Security

Looking ahead, semantic layers will evolve in three key directions:

Standardization

Common semantic models will reduce the need for custom development. We'll share entity definitions and relationship structures like we share threat intelligence today. OCSF and similar frameworks are laying this groundwork.

Automation

ML will help maintain semantic models by learning from patterns and proposing updates. Instead of manually defining every relationship, you'll validate what the system discovers. Humans will shift from creation to curation.

Distributed Intelligence

Rather than centralizing all semantic reasoning in one layer, we'll see semantic capabilities embedded throughout security infrastructure. Your EDR will understand user context, your SIEM will reason about asset relationships, your SOAR will make decisions based on business impact. The semantic layer becomes infrastructure, not a product.

The organizations that succeed won't be those with the most sophisticated semantic models, but those that solve specific problems with appropriately scoped semantic reasoning. Start with Jerry and his VPN, not with a complete ontology of your enterprise.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

This is Part 2 of the SOC Reality Check series. In Part 1, I explained how we've made incredible progress on the middle (investigation/triage) and we're improving the left (detection engineering), but this success revealed a new bottleneck: the right side of the IR cycle. Now let's talk about what it actually takes to solve it, and what becomes possible when you do.

Let me start where Part 1 ended: We're actually winning at AI-powered security operations.

The middle is getting better and better, and the left is improving. But the right side, forensics, response, and recovery, is still the bottleneck.

In Part 1, I showed you what that bottleneck looks like. But here's what I didn't talk about: what becomes possible when you solve it.

Because the right side isn't just about faster response. It's about unlocking an entirely different way of operating your SOC, one where your team actually has time to hunt for threats proactively, validate your detections continuously, and improve your security posture measurably.

This edition is sponsored by Binalyze

Investigate cyber threats in minutes

AI-powered speed. Human-driven insight.

Binalyze AIR is the forensic investigation automation platform accelerating incident response with AI precision – fast. 

Learn more at - [Link]

Why AI Alone Can't Drive Response

Let me be very clear: AI is incredibly valuable for the right side, but only if you have the forensic automation infrastructure underneath it.

I keep seeing vendors pitch "AI-driven response" and "autonomous remediation." Here's what they usually mean: automated containment like blocking an IP, quarantining an endpoint, or disabling a user account.

Here's what they don't mean: Actually collecting forensic evidence. Understanding the full scope of the compromise. Hunting for related activity across the environment. Preserving artifacts before they're overwritten.

AI can help you decide what to investigate and recommend actions. It can correlate patterns and suggest attack paths. That's the brainwork, and AI is genuinely good at it.

But AI cannot collect memory dumps before the evidence is overwritten. It can't preserve forensic artifacts across Windows, Linux, macOS, and cloud consistently. It can't ensure chain of custody. It can't guarantee complete evidence instead of just whatever happened to be available.

AI is the brain that decides what to do. Forensic automation is the hands that actually do it. You need both, and right now most organizations only have the brain.

What Forensic Automation Actually Enables

Most people think forensic automation is just about "collecting evidence faster", cutting the collection time from 4 hours to 15 minutes.

That's only 20% of the value. The other 80% is what it unlocks downstream.

You Can Finally Afford to Be Thorough

Your AI SOC platform does an amazing job. It investigates 1,000 alerts per day, closes 700 as benign, and escalates 100 as True Positive or Inconclusive. Your dashboards look fantastic.

But here's what's actually happening: Your team looks at those 100 escalated alerts and realizes manual evidence collection takes 2-4 hours per incident. With 6-8 analysts on shift, they can properly investigate maybe 10-15 incidents with complete forensics.

So what happens to the other 85? You triage, you prioritize, you make judgment calls. "This one is probably just noise, close it." "This one is non-critical, deprioritize."

You're rationing investigation capacity.

Now, imagine the math changes completely. Evidence collection happens automatically in 5-15 minutes, triggered by alert severity. By the time an analyst looks at the incident, all forensic artifacts are already collected and waiting.

Suddenly you can be thorough with everything. You're no longer choosing which threats are "worth" deep investigation, you investigate all of them.

Related to this blog, there is a podcast episode, check it out:

You Can Validate Your AI's Accuracy

Here's what keeps me up at night: How do you actually know if your AI is making correct decisions?

Of those alerts your AI closed as "benign" with 95% confidence, how many did you validate with forensic evidence? Most organizations have no idea. The AI says "95% confident," you trust it (because you don't have capacity to investigate manually), and the alert gets closed.

This is flying blind with extra steps.

With automated forensic collection, you can close this loop. AI flags True Positive → Forensics collected automatically → Human validates with evidence → Results feed back to improve AI. Without forensic automation, this loop is broken.

From Reactive Firefighting to Proactive Hunting

When your team isn't drowning in manual evidence collection, something amazing becomes possible: they can finally do proactive work that actually improves security posture.

Let me show you what I mean.

The Reactive SOC: Where Most Teams Are Stuck

Your analyst arrives at 8:00 AM, checks overnight alerts. Spends the morning manually collecting evidence, SSH into systems, running scripts, hoping logs haven't rotated. By lunch, they've collected partial evidence from a few systems. Afternoon brings more alerts, more manual collection, more coordination. By 5:00 PM, they go home exhausted.

Time spent on proactive threat hunting: zero.

The Proactive SOC: What Becomes Possible

Same analyst, different infrastructure. By 9:00 AM, they've reviewed pre-collected evidence for all overnight incidents. Five are malicious, containment actions executed. Three are false positives, feedback logged. By 10:00 AM, incidents are handled.

Now they actually have time for proactive work.

10 AM to 12 PM: Hypothesis-driven threat hunting. 1 PM to 3 PM: Detection engineering based on findings. 3 PM to 5 PM: Purple team validation exercises.

5:00 PM: They go home having actually improved security posture, not just responded to what already happened.

Real-World Example: Hunting for Glassworm

Let me walk you through a real scenario. In January 2025, security researchers disclosed Glassworm, the first self-propagating worm using "invisible code" that hit VSCode marketplaces. This is a supply chain attack targeting developer environments.

What makes Glassworm nasty: it spreads through malicious VSCode extensions that look legitimate. Malicious code is hidden using zero-width characters. Once installed, it can steal credentials, propagate through git repositories, and establish persistence.

The problem: Most organizations have zero visibility into what VSCode extensions developers install. It's not something EDR monitors closely. Developers install extensions all the time, it's normal behavior.

The Reactive Approach: Waiting for Something Bad to Happen

Someone reads about Glassworm on Twitter. "We should probably check if we're vulnerable." Added to backlog.

Two weeks later, a developer tickets IT: "My system is acting weird." The ticket sits in the queue for days, it's not high priority.

Eventually, an analyst investigates. They check EDR logs for that one system. VSCode is making unusual network connections, but is that malicious? They don't have forensic artifacts to tell. They ask the developer what extensions are installed, the developer lists five they remember, but there are actually twelve installed.

How do you investigate a VSCode extension? They're just directories of JavaScript files. Is the code malicious? This investigation is going nowhere.

Meanwhile, the worm has been propagating for three weeks through git repositories. Other developers got infected. Nobody knows the full scope.

This is what happens when you're purely reactive.

The Proactive Approach: Hunting Before It Becomes an Incident

It's Monday morning, January 20th. The security team reads about Glassworm disclosed over the weekend. Instead of adding it to backlog: "Let's hunt for this right now."

Now, you might be thinking: "Wait, can't I do this hunt with my EDR?" And you'd be partially right. Yes, your EDR has some of this telemetry, process execution logs, network connections, maybe even some file system activity if you're collecting it.

But here's where it gets complicated:

Your EDR covers the 347 Windows and Mac developer workstations with agents installed. Great. But what about the 45 Linux developer boxes that use a different EDR agent with a different console? Or the 20 cloud development environments that don't have traditional endpoints? Or the containerized dev environments that spin up and down?

You're already in three different tools.

Now let's say you start hunting in your EDR console. You query for VSCode process execution across all endpoints. Your EDR retention is 90 days if you're lucky (and paying premium for that). The Glassworm disclosure mentions extensions installed three months ago. You might be outside your retention window for the initial infection.

You find some VSCode processes making unusual network connections. Good! But to investigate the actual extension code, you need file system artifacts, the extension directories, the JavaScript files, the package.json configs. Does your EDR collect that level of disk forensics? Maybe for specific directories you've configured, but did you tell it to monitor VSCode extension folders six months ago? Probably not.

Now you want to check persistence mechanisms. You pivot to your SIEM to search Windows event logs for scheduled tasks. Oh, but you need to correlate those with the specific VSCode processes you found in EDR. You're copy-pasting process IDs and timestamps between tools.

You want to see what git repositories the infected systems accessed, because the worm propagates through git. That data might be in your network monitoring tool or proxy logs. Now you're in a fourth tool, trying to correlate timestamps again.

And let's say you find evidence of compromise. Now you need to preserve it properly for potential legal proceedings with proper chain of custody. Your EDR wasn't really designed for that, it's a detection and response tool, not an evidence management system.

This is what I mean by "might be limited and you need to jump to multiple tools."

Why Unified Forensic Automation Changes This

With forensic automation platform collecting data across all your systems, not just endpoints with EDR agents, the hunt looks different:

First query: What VSCode extensions are installed across all developer systems?

One query. One platform. Covers Windows, Mac, Linux, cloud instances, containers, everything. Results in minutes: 1,240 unique extensions across 412 total systems (not just the 347 your EDR covers).

The forensic platform has been collecting file system artifacts continuously, not just process executions, but actual extension files, configuration, modification timestamps. You can see exactly what's installed, when it was installed, and examine the actual code if needed.

Second query: What network connections are VSCode processes making?

Same platform. Network forensics correlated automatically with process data. No pivoting to another tool. You're seeing unusual patterns on 12 systems.

Third query: What persistence mechanisms were created by VSCode processes?

Same platform. Scheduled tasks, startup items, cron jobs, all correlated with the processes that created them. You don't need to manually match timestamps across three different tools.

All of this in one investigation workspace. The forensic artifacts are preserved with proper chain of custody. Retention isn't limited to 90 days, you have historical data going back as far as you need for compliance.

You find 12 compromised systems across all your development infrastructure, including three Linux boxes and two cloud instances your EDR doesn't cover.

This is why "EDR has logs" isn't the complete answer. EDR is fantastic for what it does, but it's not designed to be a comprehensive forensic investigation and hunting platform. It's designed for endpoint detection and response.

The Tool Sprawl Problem

Here's what hunting looks like in most organizations today, even with good EDR:

• EDR console: Process execution, network connections (for systems with agents)

• SIEM: Log aggregation, scheduled tasks, authentication events

• Network monitoring tools: Proxy logs, DNS, NetFlow

• Identity systems: Account access patterns, privilege changes

• Cloud security tools: Cloud workload activity

• Vulnerability scanner: Patch status, software inventory

When you hunt, you're pivoting across all of these, manually correlating timestamps and indicators. Copy-paste investigation. It works, but it's slow, error-prone, and doesn't scale when you need to hunt across thousands of systems in two hours.

Forensic automation platforms consolidate this. Not by replacing your EDR or SIEM, but by providing a unified collection and investigation layer on top of them. You're still using your EDR for real-time detection and response. But when you need to hunt or do deep investigation, you have one place where all the forensic artifacts live, properly correlated, with extended retention.

The Detection Engineering Payoff

After the hunt, the team asks: "Why didn't our detections fire for this?" They had EDR monitoring, but weren't specifically looking at VSCode extension installations or obfuscated code execution from extension directories.

New detection built: Monitor for VSCode extensions from non-verified publishers making unusual network connections or creating persistence. Tuned using the forensic data they collected, tested against the twelve confirmed infections and 335 clean systems.

This is security posture improvement in action, each hunt makes your detection coverage broader and your blind spots smaller.

The Cultural Transformation

Solving the right side requires a fundamental shift in how SOC teams think about their work.

Most SOCs operate with a reactive mindset: Alerts come in, you respond. Success = how fast you clear the queue. You're constantly behind, always catching up.

When you solve the right side with forensic automation, you can shift to a proactive mindset: You assume threats are already in your environment that your detections missed. Your job is to actively hunt for them. Success = how much you improve detection coverage and validated security controls.

This shift doesn't happen automatically. It requires leadership commitment, restructuring how analyst time is allocated (dedicating 20-30% to hunting), changing what metrics you track, and training your team on hunting methodologies.

But the organizations that make this shift? They're the ones who find supply chain attacks like Glassworm before they spread through the entire organization.

The Practical Implementation Path

Phase 1: Build the Forensic Automation Foundation

Everything depends on this. You need a system that can remotely collect comprehensive forensic evidence across all OS types, triggered automatically based on alert severity, preserving chain of custody properly.

This doesn't mean ripping out your EDR. Your EDR is still critical for real-time detection and response. But you need a forensic layer that:

• Collects deeper artifacts than typical EDR telemetry (disk forensics, memory dumps, full file system analysis)

• Covers systems beyond traditional endpoints (cloud instances, containers, systems without agents)

• Provides extended retention without the cost of expanding EDR storage

• Unifies investigation across all your security tools in one workspace

• Preserves evidence properly for legal and compliance needs

Tools like Binalyze are purpose-built for this problem. You're not replacing your SIEM or EDR, you're filling the gap between "alert fired" and "comprehensive evidence collected for hunting and investigation."

Phase 2: Free Up Analyst Time for Hunting

Allocate dedicated hunting time (20-30% of each analyst's week). Make it protected time, not "hunt when you have spare cycles." Define a hunting cadence, weekly TTP-based hunts, monthly purple team exercises, quarterly baseline deviation hunts.

Phase 3: Close the Feedback Loops

From hunting to detection engineering: What gaps exist? What new rules should be built? The Glassworm hunt should result in new detections for supply chain attacks.

From response to AI improvement: Was the AI's verdict correct? Validate with forensic evidence.

From forensics to threat intelligence: Share what you're actually seeing in the wild.

The Bottom Line: Beyond "Faster Response"

The right side isn't just about responding faster to incidents. It's about fundamentally transforming how your SOC operates.

When you solve the right side with forensic automation, the response becomes faster and the evidence is complete. But those are just the table stakes.

The real transformation is that proactive hunting becomes feasible. Your team can actually hunt for threats when intel drops, instead of hoping your detections catch it. Detection engineering gets continuous feedback. Your security posture improves measurably over time.

Yes, your EDR gives you telemetry. Yes, you can do some hunting with it. But when you need to hunt across your entire infrastructure, endpoints, cloud, containers, systems without agents, and correlate findings without pivoting through five tools, that's where forensic automation makes the difference.

This is the shift from a SOC that fights fires well to a security operation that prevents fires from starting. From reactive response to proactive defense. From hoping your detections work to validating they work and continuously improving them.

We did great work solving the middle with AI triage. We're making real progress on the left with better detection engineering. But the right side is what unlocks the SOC everyone actually wants to work in, one that hunts threats proactively and improves security posture measurably.

Don't just close alerts faster. Build a SOC that actually gets ahead of threats.

That's what the right side is really about.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

A two-part series on the new bottleneck emerging in AI-powered SOCs, and why solving investigation created a better problem to solve.

Let me start with some good news: We're actually winning at AI-powered security operations.

No, really. Hear me out.

Over the last two years, AI has genuinely transformed the middle of the IR cycle, investigation and triage. Vendors build amazing AI SOC analysts that can handle the 5Ws (Who, What, When, Where, Why), enrich alerts with context, and reach accurate verdicts in seconds instead of hours.

The middle is getting solved. AI can investigate alerts at scale, and it does it well (of course if you implement it right)

And now? The industry is shifting left. We're seeing platforms tackle data pipelines, log normalization, and even detection engineering itself. AI is helping us build better detections, optimize coverage, and reduce false positives at the source.

This is real progress. We should celebrate it.

But here's what's happening as a result of this progress: we're moving the bottleneck.

This edition is sponsored by Binalyze

Investigate cyber threats in minutes

AI-powered speed. Human-driven insight.

Binalyze AIR is the forensic investigation automation platform accelerating incident response with AI precision – fast. 

Learn more at - [Link]

The Problem We Created by Solving the Middle

When AI handles investigation and triage brilliantly, and when detection engineering improves to generate higher-fidelity alerts, you end up with a new challenge:

You now have more True Positives and Inconclusive alerts that require deep investigation and incident response.

Let me break this down with a real example:

Before AI (the old bottleneck):

• 1,000 alerts per day

• Analysts can investigate maybe 50-100 per day

• 900 alerts never get properly looked at

• Real threats hiding in the noise

• Bottleneck: Can't investigate everything

After AI in the Middle (current state):

• 1,000 alerts per day

• AI investigates all 1,000

• AI closes 700 as benign false positives

• AI flags 200 as low-confidence, likely benign

• AI escalates 100 as True Positive or Inconclusive

• Analysts now have 100 high-quality alerts that need deeper investigation

• New bottleneck: Can't properly respond to everything

See what happened? You went from "drowning in alerts" to "drowning in incidents that need forensic investigation and response."

This is actually a better problem to have, you're working on real threats now, not noise. But it's still a bottleneck.

Related to this blog, there isa podcast episode, check it out:

But Here's Where It Gets Interesting

While the bottleneck has clearly shifted right, smart organizations are discovering that forensic automation platforms can actually enhance investigation at every stage, not just the deep forensic response. 

Even during AI triage, when your AI SOC analyst is determining whether that alert is a True Positive or False Positive, having instant access to forensic artifacts can dramatically improve verdict accuracy:

• Memory analysis reveals the full process tree AI couldn't see from logs alone

• Disk artifacts show persistence mechanisms that log data missed

• Network connection history provides context that makes "Inconclusive" → "True Positive"

This means fewer alerts stuck in "Inconclusive" limbo and more confident verdicts earlier in the process. The forensic infrastructure you build for deep investigation also makes your AI triage more effective.

What "Deep Investigation and Response" Actually Means

When your AI SOC analyst escalates a True Positive or marks something as Inconclusive, here's what actually needs to happen:

The AI hands you a verdict:

TRUE POSITIVE: Lateral movement detected

User: john.doe

Source: WORKSTATION-047  

Target: DC-01 (admin share accessed)

Confidence: HIGH

MITRE: T1021, T1570

Recommendation: Immediate forensic investigation and containment

Great! Now what?

Your analyst (or IR team) needs to:

1. Collect forensic evidence from affected systems
   

   • Memory dumps before artifacts are overwritten

   • Disk analysis for persistence mechanisms

   • Process execution history

   • Network connections and data transfers

2. Determine blast radius

   • What other systems did this user access?

   • Are there signs of lateral movement elsewhere?

   • What data might have been accessed or exfiltrated?

3. Preserve evidence for potential legal/compliance needs

   • Chain of custody documentation

   • Timeline reconstruction

   • All artifacts properly stored and indexed

4. Execute containment according to your IR playbook

   • Isolate affected endpoints

   • Disable compromised accounts

   • Block malicious IPs/domains

   • (All with proper approvals and governance)

5. Coordinate with stakeholders

   • IT for system access and changes

   • Legal for compliance and evidence handling

   • Management for business impact decisions

   • External parties if breach notification required

And here's the kicker: most of this is still manual, slow, and inconsistent.

Why the Right Side Became the New Bottleneck

As you solve the middle (investigation/triage) and improve the left (detection engineering), you're naturally generating more high-quality incidents that require the right side of the IR cycle to work properly:

Investigation → Containment → Eradication → Recovery → Lessons Learned

But the right side hasn't kept pace with the middle and left. Here's what's still broken:

1. Forensic Evidence Collection is Manual and Slow

What actually happens:

• Analyst gets Alerts that requires further investigation at 10:00 AM

• Starts trying to collect forensic evidence at 10:15 AM

• Realizes they need to SSH into multiple systems(or use your EDR)

• Discovers some logs have already rotated out (oops)

• Manually pulls what's available from EDR (if the agent is installed)

• Tries to get memory dumps (complicated, requires specific tools and permissions)

• By 2:00 PM, maybe has incomplete forensic evidence

The problem:

• Evidence gets overwritten or rotated before you can collect it

• Collection process is different for every analyst ("tribal knowledge")

• Manual processes don't scale when you have 20 incidents per day instead of 5

2. IR Playbooks Aren't Executable

Remember my blog series on playbooks? Most IR playbooks are PDFs or wiki pages that say things like:

Lateral Movement Response:

1. Isolate affected systems

2. Collect forensic evidence

3. Determine scope of compromise

4. Reset credentials

5. Document findings

This is guidance for humans, not executable automation.

When your AI SOC solution escalates 20 True Positives per day, you can't manually execute these steps 20 times. You need:

• Automated forensic collection triggered by alert severity

• Standardized evidence preservation

• Orchestrated response actions with proper governance

• Consistent execution regardless of who's on shift

3. Response Actions Require Too Much Coordination

Even when you know what needs to be done, executing it requires:

• Multiple approval chains (IT, Security, Management)

• Coordination across teams (Security, IT, DevOps)

• Navigating change management processes

• Fear of business disruption from containment actions

So containment that should happen in minutes takes hours or days because you're stuck in Slack threads and email chains getting approvals.

The Shift Map: Where We Are and Where We're Going

Let me map this against the SecOps AI Shift Map I introduced in my previous blog:

Left Side (Detection & Data):

• ✅ AI is starting to help here

• ✅ Better detection engineering with AI assistance

• ✅ Improved data pipelines and normalization

• Status: Progress being made

Middle (Investigation & Triage):

• ✅ AI SOC analysts handle this brilliantly

• ✅ Enrichment, 5Ws, verdict reached in seconds

• ✅ False positives filtered out effectively

• Status: Progress being made

Right Side (Response & Recovery):

• ❌ Forensic evidence collection still manual

• ❌ IR playbooks not machine-executable

• ❌ Response actions require too much human coordination

• ❌ Recovery and lessons learned feedback loops broken

• Status: This is the new bottleneck

Why This Matters: The Investigation Debt Problem

Here's a concept I don't hear talked about enough: Investigation Debt.

Investigation Debt is what accumulates when you have True Positives or Inconclusive alerts that you can't properly investigate with complete forensic evidence.

Every time you:

• Close a True Positive without collecting full forensic artifacts

• Skip deeper analysis because evidence collection is too manual

• Move on because "we don't have time to do full forensics on everything"

• Accept "the logs rotated out" as an answer

You're accumulating Investigation Debt.

And here's what makes this dangerous: that lateral movement alert you couldn't fully investigate three months ago might have been your initial compromise. But you didn't collect the forensic evidence, logs rotated, the attacker cleaned up, and now you're doing incident response in the dark.

The irony is that better AI in the middle makes Investigation Debt more visible.

Before AI, you had so much noise you didn't even know which alerts were real. Now AI is flagging possible True Positives for you, and you're realizing: "We don't have the capacity to properly respond to all of these."

What "Solving the Right Side" Actually Requires

So what does the right side need to keep pace with the middle and left?

1. Automated Forensic Evidence Collection

When a True Positive alert fires, evidence collection should happen automatically, not manually:

• Memory dumps captured before artifacts are overwritten

• Disk forensics collected immediately

• Process trees and network connections preserved

• All artifacts time-stamped and stored with proper chain of custody

This needs to be triggered by alert severity, not analyst memory.

2. Response Orchestration with Governance

Response actions need to be automated but with proper safety rails:

• Approval workflows for high-impact actions

• Blast radius checks (don't isolate 500 endpoints because one script failed)

• Rollback procedures if containment causes problems

• Audit trails of who approved what and when

This is where SOAR was supposed to help, but as I wrote in my Shift Map blog: we failed at SOAR not because the tech was bad, but because our processes were a mess.

The Path Forward: Shift Right

Here's where we are:

✅ Middle is getting solved - AI handles investigation and triage brilliantly

🔄 Left is improving - Detection engineering and data pipelines are getting AI assistance

❌ Right is the bottleneck - Forensic collection, response execution, and recovery are still manual

The solution isn't to slow down the middle or left. The solution is to build the right side infrastructure to match.

That means:

• Forensic automation platforms that collect evidence at machine speed

• Machine-executable IR playbooks that respond consistently

• Response orchestration with appropriate governance and safety rails

• Feedback loops that actually work (lessons learned → detection engineering)

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

A Quick Note Before We Start

When I wrote about how AI transforms detection engineering from narrow precision to broad coverage, I originally planned this as Part 2 of that series. The logic made sense: Part 1 shows how AI enables comprehensive detections, Part 2 would explain how to handle the resulting alert volume.

But as I started writing, I realized this piece actually belongs with a different blog I published a couple months ago: "Why SOC Analysts Ignore Your Playbooks".

Here's why: The detection engineering blog is about what you can now detect. This blog is about how you investigate what you detect. And the investigation problem traces back to the same root causes I identified in the playbooks blog: broken processes, ignored documentation, and tribal knowledge that walks out the door when your best analysts leave.

So consider this Part 2 of the playbooks series, not the detection engineering one. Though honestly, they're all connected; you can't deploy comprehensive detections without machine-executable investigation procedures, and you can't build those procedures if your playbooks are PDF documents nobody follows.

If you haven't read Part 1 yet, start there. It explains why playbooks are broken and how to fix them by coupling them with detections. This post takes that foundation and shows you how to transform those playbooks into logic that AI can actually execute.

Let's dive in.

The Problem We Thought We Solved

In Part 1, I argued that the biggest mistake in SecOps is shipping detections without playbooks. That shadow detection sitting in your SIEM that nobody understands? That's the root cause of your broken feedback loop, your alert fatigue, and your analyst burnout.

The fix seemed straightforward:

1. Build the playbook when you build the detection

2. Use GenAI to draft it in seconds

3. Automate the steps you can

4. Let AI learn from analyst behavior for the rest

And for human analysts, this works. You now have documented procedures, analysts can follow them (or at least reference them when needed), and new joiners can learn how your organization operates.

But then you try to hand these playbooks to an AI agent to execute autonomously.

And it all falls apart.

This edition is sponsored by Prophet Security

Put every alert through a complete investigation.

Prophet AI pulls the right context, follows a reproducible line of questioning, and returns a clear determination with linked evidence and an audit trail. Analysts stay in control while investigations finish faster and hold up under review. 

See it on your data. Request a demo at prophetsecurity.ai

Why Your "Fixed" Playbooks Still Break AI

Let's take a typical playbook that follows my advice from Part 1. It's not a dusty PDF,it's embedded with the detection, it's maintained, and it actually reflects how your team works:

Suspicious Login Investigation Procedure

When you receive a suspicious login alert, follow these steps:

1. Check whether the login location matches the user's typical locations

2. Review their login history for the past 30 days in your identity provider

3. If the location is unusual, verify whether the user recently traveled

4. Check if the device fingerprint is recognized

5. If multiple red flags exist, contact the user's manager

6. If you cannot confirm legitimacy within 2 hours, force password reset

This is a GOOD playbook by Part 1 standards. It's clear, actionable, tied to a specific detection. An experienced analyst knows exactly what to do.

But hand this to an AI agent, and watch what happens:

What exactly is an "unusual location"?

• 50km away? 500km? Different country? Different continent?

• What if the user works remotely and travels frequently?

• What if they're using a VPN that makes location data unreliable?

What constitutes "multiple red flags"?

• Two flags? Three? Which combinations matter more?

• Is unusual location + recognized device better or worse than normal location + unrecognized device?

• How do you weight these factors?

What if your identity provider is temporarily unavailable?

• Does the AI wait? For how long?

• Does it proceed without that data? How does that affect its confidence?

• Does it automatically escalate?

When should AI act autonomously vs. escalate to a human?

• Can it force a password reset on its own? For which user types?

• What if the "suspicious" login is the CEO accessing email at 2 AM?

Human analysts handle this ambiguity through experience, context, and judgment. They know that "unusual" for the CEO is different than "unusual" for a contractor. They know which data sources to trust. They adapt when systems are down.

AI doesn't have that context unless you give it explicitly.

The Missing Layer: Machine-Executable Logic

This is what I've realized since Part 1: there's a gap between what humans need and what AI needs.

Humans need:

• Clear guidance on what to investigate

• Reference points for common scenarios

• Flexibility to adapt based on context

• Your Part 1 playbooks solve this

AI needs:

• Explicit decision logic with no ambiguity

• Quantifiable thresholds and confidence scores

• Complete coverage of edge cases and fallbacks

• Structured entity resolution across data sources

• Your Part 1 playbooks don't solve this

So we need a new layer. Not to replace human-readable playbooks, but to sit underneath them.

Let me show you what this looks like.

The Same Playbook, But Machine-Executable

Here's how that suspicious login investigation transforms when designed for AI execution:

yaml
playbook: suspicious_login_investigation
version: 2.3
owner: identity_security_team

# Data source requirements with structured fallbacks
required_data_sources:
  okta_logs:
    retention_days: 30
    required_fields: [user_id, src_ip, location, device_fingerprint, mfa_status]
    unavailable_action:
      type: cap_max_confidence
      max_confidence: 0.80
  
  vpn_logs:
    retention_days: 7
    unavailable_action:
      type: confidence_penalty
      delta: 0.10

# Entity resolution: how to connect data across systems
entity_resolution:
  user_identity:
    canonical_identifier: email
  
  crowdstrike_endpoint:
    steps:
      - {action: map, from: alert.email, to: active_directory.userPrincipalName}
      - {action: get, field: active_directory.lastLogonComputer}
      - {action: map, from: active_directory.lastLogonComputer, to: crowdstrike.hostname}
      - {action: get, field: crowdstrike.AID}
    fallback: {strategy: use_cache, max_age_hours: 24}

# Confidence scoring: weights sum to 1.0
confidence_scoring:
  method: weighted_sum
  normalize_weights: true
  
  factors:
    - name: location_deviation
      weight: 0.30
      scoring:
        - {when: "distance_km < 50 AND time_plausible", score: 0.80}
        - {when: "distance_km >= 500 OR impossible_travel", score: 0.20}
    
    - name: mfa_status
      weight: 0.25
    
    - name: device_recognition
      weight: 0.25
    
    - name: time_of_day_anomaly
      weight: 0.15
    
    - name: concurrent_activity
      weight: 0.05

# Complete decision ranges (no gaps: 0.0-1.0 covered)
decision_thresholds:
  auto_close_benign: {range: [0.95, 1.00], action: close_with_docs}
  auto_close_fp: {range: [0.90, 0.95], action: close_and_tune_detection}
  low_risk_action: {range: [0.75, 0.90], action: require_mfa_reauth}
  escalate_medium: {range: [0.60, 0.75], action: analyst_review, sla_hours: 4}
  escalate_high: {range: [0.40, 0.60], action: analyst_review, sla_hours: 2}
  escalate_critical: {range: [0.00, 0.40], action: incident_response, sla_minutes: 30}

# Safety rails for autonomous actions
governance:
  suspend_account:
    approval_required: true
    two_person_rule: true
    blast_radius_check: true
  
  isolate_endpoint:
    approval_required: true
    blast_radius_limit: 5

# Audit with privacy controls
audit_trail:
  capture: [data_sources_queried, confidence_scores, reasoning_chain, action_taken]
  privacy: {redact_pii: true, respect_residency: true}
```

See the difference?

No ambiguity. "Unusual location" becomes distance_km >= 500 OR impossible_travel with a confidence score of 0.20.

No gaps. Every possible confidence score from 0.0 to 1.0 maps to a specific action.

Explicit fallbacks. If Okta is down, max confidence caps at 0.80. If VPN logs are unavailable, confidence drops by 0.10.

Measurable weights. Location deviation is weighted 0.30, MFA status 0.25, device recognition 0.25,you can validate these against historical outcomes.

Clear governance. AI can suggest account suspension, but requires human approval with two-person rule.

This isn't documentation for humans to read. It's executable logic for AI to run.

Connecting Back to Part 1: The Full Picture

Remember in Part 1 when I talked about two approaches to fixing playbooks?

• Approach 1: Build playbooks during detection engineering

• Approach 2: Let AI observe analysts and generate playbooks from behavior

Both of these still apply. But now I'm adding a critical step:

Approach 1 Extended: Detection Engineering → Machine Logic

1. Build human-readable playbook with detection (Part 1)

2. Identify which steps can be automated

3. Transform those steps into machine-executable logic (Part 2)

Approach 2 Extended: Behavioral Learning → Quantified Rules

1. Let AI observe how analysts investigate alerts (Part 1)

2. Generate initial playbook from behavior patterns

3. Codify the decision logic into structured, quantifiable rules (Part 2)

The insight from Part 1 still holds: analysts going by "instinct" have built better processes in their heads.

The challenge in Part 2 is: how do we capture that instinct as explicit, measurable logic that AI can execute?

This is where those analysts who ignore playbooks actually become your most valuable asset. Watch how they handle edge cases. Ask them why they made each decision. What data points did they weigh most heavily? When did they escalate vs. auto-close?

That tribal knowledge you were trying to capture in Part 1? Now you're quantifying it for Part 2.

What This Enables (Beyond Just Automation)

When I published Part 1, the focus was on fixing broken processes. If playbooks are ignored, your feedback loop breaks, your detection tuning stops, and analyst burnout accelerates.

But machine-executable playbooks unlock something bigger: measurable, improvable AI decision-making.

In Part 1, I mentioned the vanity metrics problem, everyone obsesses over alert closure times, but nobody tracks whether AI is making correct decisions.

Machine-executable playbooks make real metrics possible:

AI Decision Accuracy

• Of benign closures, how many were actually benign?

• When AI says 95% confident, is it right 95% of the time?

• Which alert types show the most false positives from AI triage?

Learning Over Time

• Does accuracy improve week over week?

• Do analyst overrides result in playbook improvements?

• Is the system getting better at handling your specific environment?

Drift Detection

• Is AI accuracy degrading as your environment changes?

• Which confidence factors show the most drift?

• Are updates keeping pace with infrastructure changes?

You can't measure any of this with ambiguous playbooks that analysts interpret differently on every shift. But with explicit, structured logic? You can track every decision, validate every confidence score, and continuously improve.

This is the feedback loop Part 1 was trying to fix, now operating at AI speed.

The Six Requirements for Machine-Executable Playbooks

Based on everything I've learned building these for real environments, here's what actually matters:

1. Explicit Decision Logic (No Ambiguity, No Gaps)

Your decision thresholds must cover every possible confidence score from 0.0 to 1.0. No "if unusual, escalate" nonsense. Define EXACTLY what triggers each action.

❌ Bad: If suspicious, escalate to analyst

✅ Good:

2. Data Topology and Entity Resolution

Your AI needs to know how to connect data across your specific tool stack. How do you go from alert email → Active Directory → CrowdStrike endpoint ID?

This was implicit tribal knowledge in Part 1's world. In Part 2, it must be explicit:

3. Confidence Calibration (Validated Against Outcomes)

Remember in Part 1 when I said the feedback loop is broken because analysts ignore the step that says "send false positives back to detection engineering"?

Machine-executable playbooks fix this by requiring outcome validation:

• If AI says 90% confident benign, track how often it's actually benign

• Adjust confidence weights based on historical accuracy

• Recalibrate when you detect drift

The confidence scores must be trustworthy, not aspirational.

4. Reproducible Metrics and Evaluation

When a vendor claims "improves coverage from 50% to 95%," ask: "95% of what, measured how?"

With machine-executable playbooks, you can test against:

• Historical incidents with known outcomes

• Purple team exercises covering MITRE ATT&CK techniques

• Adversary emulation with documented expected results

Define variant coverage as: (# of distinct attack technique variants correctly triaged) / (total curated variants in test set)

This separates real capability from marketing claims.

5. Safety Rails and Governance

In Part 1, I talked about how deterministic automation needs approval workflows. The same applies here, but even more critically.

AI Agents handle: Investigation, enrichment, triage decisions, confidence scoring, and recommending next steps

Deterministic automation handles: Response execution with defined approval workflows, rate limiting, blast radius checks

Why this matters: AI reasoning excels at weighing ambiguous evidence during investigations. However, response actions need predictability, auditability, and fail-safe controls.

Governance controls:

• Confidence thresholds determine when AI can auto-close vs. escalate

• Escalation criteria define what triggers human involvement

• Override tracking captures when analysts disagree with AI (this feeds back into learning)

6. Complete Observability and Transparency

In Part 1, I said the problem with playbooks is they're ignored and never maintained. Part 2 solves this by making every AI decision traceable:

• Complete audit trails showing data queried, confidence calculated, thresholds applied

• Reasoning chains explaining every decision step

• Metrics tracking accuracy, drift, and performance over time

• Testing frameworks for validating playbook changes before deployment

Black boxes are unacceptable. If you can't see how AI reaches conclusions, you cannot trust it, improve it, or measure its effectiveness.

This is how you prevent Part 2 from becoming Part 1 all over again,broken processes that nobody maintains.

Two Paths to Implementation (Revisited from Part 1)

In Part 1, I outlined how to build playbooks alongside detections. Now let's extend that with two approaches to machine-executable logic:

Route 1: Customizable Intelligence

The approach: Build your own machine-executable playbooks for your specific environment. You control procedures, entity mappings, confidence weights, and thresholds.

Best for: Mature security operations with established procedures, engineering resources to invest, and unique tool stacks requiring customization.

What you get: Maximum control to encode your specific tribal knowledge and operational context,the "instinct" your best analysts use.

What it requires: Security engineering resources to build, test, and maintain playbooks over time. This is the formalization of the behavioral learning I mentioned in Part 1.

Route 2: Out-of-the-Box Intelligence

The approach: Pre-trained AI with built-in investigation procedures based on security best practices and collective intelligence across many deployments.

Best for: Faster time-to-value, teams lacking resources to build custom playbooks, organizations wanting proven procedures.

What you get: Working baseline from day one, proven procedures, faster deployment.

What you can customize: Thresholds, escalation criteria, and specific organizational context as you learn what works.

The Non-Negotiable Requirement

Regardless of path, transparency is mandatory.

Remember the RPA problem I mentioned in Part 1? RPA solutions failed in cybersecurity because the tech wasn't there and the environment was too unpredictable.

AI solves the unpredictability through non-deterministic reasoning. But only if you can:

✓ See the investigation logic and understand what the AI does
✓ Trace the complete reasoning chain from alert to decision
✓ Audit all decisions with confidence scores and data sources
✓ Modify procedures as your requirements change
✓ Measure effectiveness with accuracy, learning, and drift metrics

Critical questions to ask any AI SOC platform:

• Can I see the investigation logic being applied?

• How are confidence scores calculated?

• What happens when data sources are unavailable?

• How do you handle entity resolution across my specific tools?

• Can I test changes before deploying them?

• How do you measure and track accuracy over time?

Without answers to these, you're just automating chaos faster.

The Bottom Line: Completing the Transformation

In Part 1, I showed you why playbooks are broken and how to fix them by coupling them with detections. That solves the human problem.

But Part 2 is the necessary evolution: those human-readable playbooks don't work for AI execution.

Machine-executable SOPs bridge the gap between broader detections and manageable operations. They're what allow AI to triage thousands of alerts with the same quality your best analyst applies to dozens.

This isn't better documentation. It's a fundamental shift in how security knowledge gets operationalized,moving from ambiguous human guidelines to explicit, testable, measurable logic that can be continuously improved.

The connection to Part 1:

• Analysts ignore playbooks → Build them with detections

• Detections without playbooks break feedback loops → Automate what you can

• Analysts go by instinct → Capture that tribal knowledge

• Now codify that knowledge into logic AI can execute → Measure and improve systematically

The six requirements that matter:

1. Explicit decision logic with complete coverage (no gaps, no ambiguity)

2. Data topology and entity resolution (connecting data across your specific systems)

3. Confidence calibration (trustworthy thresholds validated against outcomes)

4. Reproducible metrics (defensible claims about coverage and accuracy)

5. Safety rails (governance preventing business disruption)

6. Complete observability (transparency enabling trust and improvement)

Whether you build custom playbooks or start with proven out-of-the-box intelligence, these requirements don't change. The platform must provide them, and you must be able to validate they're working in your specific environment.

Without Part 1, your processes stay broken.

Without Part 2, you're automating chaos faster.

With both, you're building a security operation that measurably improves over time.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Part 1 of 3: The Detection Coverage Problem and How AI Solves It

Your SOC processes ten thousand alerts daily. Your detection engineer just wrote a brilliant new rule detecting lateral movement via WMI, but here’s what happens next:

They look at the alert volume and realize it generates two hundred potential hits per day. They know your team can realistically investigate maybe twenty alerts per day for this detection type, so they start making the rule more restrictive. They add filters, raise thresholds, and narrow the scope until the alert volume drops to something manageable.

In doing so, they’ve just created a blind spot.

Those one hundred and eighty alerts they filtered out might contain real threats, but your process design forced them to choose between overwhelming the team and potentially missing attacks.

This is the fundamental problem we need to solve. Your processes were designed for human-in-the-loop execution, and that constraint is now the bottleneck strangling your security effectiveness.

This edition is sponsored by AiStrike

AI SOC Done Right!

AI SOC Intelligence Fabric that unifies your data, accelerates investigations, and orchestrates intelligent response

Transform your SOC with composite AI that arrives pre-trained and ready to work. Enable small teams to operate like enterprise SOCs while giving enterprises state-of-the-art incident response capabilities.

What changed today

Today's AI wave is not a plug-and-play upgrade for security operations. Just as the shift to cloud and SaaS forced organizations to realign processes, roles, and governance, the AI wave demands a full reboot of your people-process-technology stack.

This isn't about adding a new tool to your existing workflows. This is about fundamentally rethinking how security operations function when you remove the human throughput constraint from the equation.

The Research Backs This Up

Recent research reveals an uncomfortable truth: data quality predicts success more than raw technology capacity, and process design often outweighs management intent in driving integration [1]. Meanwhile, cybersecurity researchers are exploring human-AI co-teaming models in SOCs, stressing the need for dynamic autonomy, trust calibration, and feedback loops in operational workflows [2].

The crux: Dropping AI into a rigid SOC is like installing a jet engine on a cart with square wheels. The power is there, but the system isn't designed to harness it.

What's Really Changing

Every major technology wave forces security teams to renegotiate the relationship between people, process, and technology:

• Cloud era: Reinvented access models, monitoring pipelines, identity governance

• SaaS era: Adapted to distributed ownership and ephemeral infrastructure

• AI era: Must handle systems that don't just observe, they act, decide, and recommend

The challenge isn't visibility anymore. It's an agency. AI systems don't just help us monitor threats; they investigate, triage, and recommend actions. That shift means SOC processes can't remain static, checklists written for human cognition. They must become machine-executable logic that adapts to model confidence, context, and risk.

The Brutal Trade-Off: Killing Your Detection Coverage

Traditional detection engineering operated under constraints that forced you to sacrifice coverage for operational feasibility. Let me show you what this looks like in practice.

The Typical Detection Engineering Process

Here's how it actually works:

1. Developp hypothesis about a threat you want to detect

2. Build a detection rule to identify that behavior

3. Test against your environment to see alert volume

4. See 100 alerts per day 😰

5. Realize your team can only handle 20 alerts per day

6. Make the detection more restrictive (add filters, raise thresholds)

7. Deploy detection that catches 40% of attack variants instead of 90%

You weren't optimizing for security effectiveness. You were optimizing for operational survival.

Understanding the Funnel of Fidelity

Zack Allen in the Detection Field Manual #3 talks about detection efficency concept of the Funnel of Fidelity (introduce by Jared Atkinson back in 2019)  to describe this exact problem[3]: massive data volume at the top, limited analyst capacity at the bottom. Every alert that survives the funnel consumes human focus, creating an inherent trade-off between comprehensive detection and operational sustainability.

This creates a dangerous dynamic. You might achieve eighty percent detection coverage, meaning your rules can theoretically identify eighty percent of relevant security events in your environment. However, analyst capacity constraints mean you can only thoroughly investigate fifty or sixty percent of the alerts those detections generate.

Your effective security coverage isn't 80%, it's the 40-50% that actually receives quality investigation.

The Attacker's Advantage

This coverage gap becomes an exploitable vulnerability. Attackers need only operate in the 20-30% of alert volume that your team doesn't have the capacity to investigate. They can:

• Generate low-level alerts that get suppressed automatically

• Operate during high-volume periods when your team is overwhelmed

• Use techniques that generate alerts your team habitually ignores due to high false positive rates

The gap between what you detect and what you investigate is where attackers live.

How AI Changes the Detection Engineering Equation

When investigation capacity increases from 20 alerts per analyst per day to thousands of alerts per AI agent per day, everything changes. You can finally deploy the detections you always wanted to build.

Before vs. After: The Transformation

The Key Shift

With AI, you move from precision-optimized detection to coverage-optimized detection.

Precision-optimized (old way):

• Question: "How can I make this detection narrow enough to be sustainable?"

• Result: Restrictive filters, high thresholds, missed attack variants

• Coverage: 30-40% of the actual threat landscape

Coverage-optimized (new way):

• Question: "How can I make this detection broad enough to catch all variants while maintaining signal quality?"

• Result: Comprehensive coverage, AI handles triage burden

• Coverage: 85-95% of the actual threat landscape

The detection engineer's job transforms completely. Instead of adding restrictive filters to reduce volume, she focuses on adding context that helps the AI make accurate disposition decisions. Instead of tuning for low volume, she tunes for high recall, knowing the AI can handle the resulting triage burden.[4]

Understanding the Alert Categories

(Updated section : Thank you Nathan Eades for the feedback)
When we talk about the triage burden, we're actually dealing with two distinct categories:

Benign Alerts: The detection is working correctly; it identified the behavior it was designed to catch. But the activity is legitimate, authorized, or expected.

• Example: Your lateral movement detection correctly flags WMI activity, but it's authorized IT maintenance during a change window

• Problem: Requires context to distinguish legitimate from malicious

False Positives: The detection is firing incorrectly due to overly broad rules or environmental noise.

• Example: Your detection fires on normal admin behavior because it doesn't account for privileged user patterns

• Problem: The detection rule itself needs tuning

Traditional SOCs struggled with both:

• Benign alerts required manual context gathering (check change tickets, verify with user, confirm authorization)

• False positives required detection tuning, but that tuning often meant narrowing the rule and missing real threats

AI handles both categories intelligently:

• For benign alerts: AI gathers context automatically (change windows, user roles, business justification) to determine legitimacy

• For false positives: AI identifies systematic patterns and suggests detection improvements

The result: You can deploy broader detections because AI can distinguish between malicious activity, benign activity, and false positives at scale.

The Transformation in Detection Philosophy

This isn't just about automation making things faster. It's a fundamental shift in how you approach detection engineering.

Traditional Detection Engineering

Guiding questions:

• Will this detection generate too many alerts?

• Can our team handle the volume?

• How can I make this more restrictive without losing too much coverage?

Optimization goal: Operational sustainability

Trade-off: Coverage sacrificed for precision

Result: Narrow detections that miss attack variants

AI-Enabled Detection Engineering

Guiding questions:

• Does this detection catch the full breadth of attacker behavior?

• What context does AI need to make accurate triage decisions?

• How can I optimize for recall without sacrificing signal quality?

Optimization goal: Security effectiveness

Trade-off: Let AI handle triage burden, focus on coverage

Result: Broad detections that catch attack variants while maintaining a manageable analyst workload

Key Metrics That Change

Traditional metrics focused on volume management:

• ✓ Alerts per day per detection

• ✓ Analyst handling capacity

• ✓ Alert-to-incident ratio

New metrics focus on coverage and learning:

• ✓ Detection recall: Of all malicious events, how many did we catch?

• ✓ AI triage accuracy: Of AI's auto-close decisions, what percentage are correct?

• ✓ Analyst amplification: How many alerts can each analyst effectively handle with AI assistance?

• ✓ Feedback utilization: Is analyst feedback improving AI accuracy over time?

What This Means for Your SOC

The transformation from narrow, precision-focused detections to broad, coverage-optimized detections has implications that ripple through your entire security operations:

For Detection Engineers

New responsibilities:

• Build comprehensive detections without volume anxiety

• Add context and enrichment logic to help AI triage

• Focus on recall and coverage rather than precision and volume

• Monitor AI triage performance and tune based on feedback

Time allocation shift:

• Less: Manual alert triage to validate detection quality

• More: Detection development, coverage expansion, AI tuning

For SOC Analysts

New workflow:

• Receive 20-30 pre-investigated cases per day instead of 200+ raw alerts

• Each case includes a full context gathered by AI

• Focus on judgment call and let the AI do the data gathering

• Provide feedback that improves AI over time

(updated section, thank yo,u Roger W. Roberts, for the feedback)

For Security Outcomes

Coverage improvement:

• From: 40-50% effective coverage (detect 80%, investigate 50%)

• To: 75-80% effective coverage (detect 80%, investigate 95%)

The Bottom Line

The shift from playbooks to agentic systems will be messy but inevitable. AI is pulling SOCs from static logic toward adaptive, self-improving systems. If the cloud era abstracted infrastructure, the AI era abstracts decision-making. Our processes must now teach machines how to operate within boundaries, not just describe what humans should do. That's not automation. That's architecture.

But here's the critical insight: This only works if you redesign your processes to take advantage of it.

Deploying broader detections into your current manual triage process just creates a bigger backlog. You need to transform how you handle the resulting alerts. That's where machine-executable investigation procedures come in.

Coming in Part 2: From PDF Playbooks to Machine-Executable Logic

Broader detection coverage only works if your investigation procedures can handle the volume. In Part 2, we'll explore the transformation that makes this possible:

You'll learn:

• Why your current SOPs don't work for AI (and what to do about it)

• How to convert human-readable playbooks into machine-executable logic

• A complete example: Suspicious login investigation (before and after)

• How does this transformation change the coverage funnel from 50% to 100% triage

• What this means operationally for your SOC team

The process tof ransformation is just as critical as technology. Get the SOP design wrong, and your AI will be making decisions based on incomplete or inconsistent logic. Get it right, and you unlock comprehensive coverage that was previously impossible.

Next week, we'll show you exactly how to do it.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

❗️News ❗️

I have been running the cybersec-automation blog for over two years now, and I decided it is time for a change.

So, I'm launching SecOps Unpacked, which is a sort of rebranding or simply expanding on what I was already doing.

While automation remains a core focus, I realised that security practitioners need more than just automation insights; they need independent research, practical frameworks, and honest vendor analysis.

I went with the idea of unpacking the security operations problems, exploring real solutions, and sharing what actually works in the field.

Some additions that you will notice are the Market research section as well tooling section (where I started a GitHub repo and soon will be sharing more Open Source tools)

I will keep my blog on the old domain until I figure out how to re-index everything and apply proper redirects.

Also, don't forget to check out the latest section, where I have a tracker of the AI SOC and Automation vendors (in-depth analysis coming soon)!

SecOps Unpacked - Research & Analysis for Security Practitioners

Unpacking SecOps problems. Exploring solutions. Sharing what actually works. Research-focused content for security practitioners and SOC engineers.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Based on what I've been hearing from the cybersecurity community, many of you are asking what the core capabilities of an AI SOC platform are that you should actually be looking at. A while back, I put together a blog that went through some of the high-level steps for implementation. This time, I want to deep-dive into the core capabilities as I see them.

You've probably seen my "AI SOC Shift" maps and graphics. They do a pretty good job of outlining the core pillars these platforms are targeting. It's important to see if they relate to your problems and to understand what type of platform you really need.

First, I’ll discuss the different implementation styles. Then, we’ll discuss data ingestion and what it looks like for AI SOC platforms. Finally, we'll cover the super important knowledge graph/DB. We'll also discuss investigations and, in the end, the response and feedback loop.

This edition is sponsored by Crogl

Built for analysts. Powered by your data. Private design

▪️Faster investigations - Cut Mean Time to Investigate (MTTI) by over 60%.
▪️Analyst-first design - Augments, never replaces. Built for real SOC workflows.
▪️Operational anywhere - Cloud, on-prem, and air-gapped environments.
▪️Customer-managed AI - Privacy, control, and compliance built in.
▪️Knowledge Engine - Automate triage, collects evidence, and documents everything.

CISO/SOC/MSSP pros, we’d love to meet you.

Implementation Styles

From what I’ve seen, there are a few types of implementations out there. We can start with the classic options like On-prem, Bring Your Own Cloud (BYOC), and SaaS. But what’s even more interesting is that some of these platforms now offer the option to bring your own LLM as well. There are some vendors moving into this space (check out the Vendor Spotlight section if you're interested).

I think this is going to be really relevant for large enterprises. Many of them are starting to develop their own internal LLMs and will want to use them. So, it’s a good sign that we're seeing vendors come up with capabilities that are truly enterprise-ready.

https://softwareanalyst.io/reports/ai-soc-industry-wide-report-2025

Data Ingestion

Here we have a few options as well. This is based on the vendors I’ve seen (demo, trial, or implemented), which is a bit over 30 in the AI SOC and next-gen security automation space.

First, you have the platforms that work just on top of your detections. These are the most common. They live on top of your detection layer, which can be a mix of SIEM, EDR, XDR, ITDR, Email Protection, and many others. What's important to consider here is what kind of tech they support. In my view, if it’s just about detection tech and excludes the SIEM, I want it to cover all my detection tools. Otherwise, you’re left with stuff you still need to handle manually or with your SOAR process.

For the platforms that also ingest from a SIEM, you should know that not all of them support every type of detection. This means many of the custom or unique detections you’ve built in your environment might not be supported, creating a gap.

Then there are others that live mainly on top of a SIEM or SOAR. This means you ingest everything you can into your SIEM, and the AI SOC platform takes any detection alert that comes out of it. The advantage here is that you can get really good coverage, as good as your SIEM allows. I've tried both implementations, and this one usually provides better results, assuming you can afford to get all your data into the SIEM.

One last thing here: some platforms can analyze your detections and come back with recommendations. Others will just ingest them and don’t put much effort into understanding why a rule runs in a specific way. For me, it's pretty important that the AI SOC platform tries to understand the logic of the rule, not just the alert it produces.

The Brains: Knowledge Graph/DB (Enrichment and Context)

This is a key component for any AI SOC solution. See it as the smart enrichment and knowledge automation engine that should go out and grab all the information it needs to understand an alert better.

From what I’ve seen, some platforms can connect to various tools that provide this information, from EDR, IAM, and Case Management to IaaS, Code Repos, and external threat intel feeds. Some might even give you some of that TI for free if you don’t want to bring your own.

I think this is critical for two reasons. First, to understand any alert, you need internal context. Think about whether the platform can ingest all the knowledge from your team, that means past incidents from Jira, wikis on Confluence or Notion, or documents on SharePoint. If you’re comfortable with it, see if they can even pull from specific IM channels where the team discusses cases.

Second, it's key to be able to connect to your internal case management solution like ServiceNow or Jira, or even GitHub repos. This allows the AI to check for relevant tickets or change requests related to the activity it's seeing.

So, put a lot of effort into evaluating this part. Make sure you can connect the tools you have so the platform can learn from your environment.

The Investigation Engine

This part can go into so many details that I'll probably do a follow-up blog on it. You can also check out a previous article where I walk through handling an EDR alert.

There are many different ways this is done, and many AI SOCs have their own method for stitching data together. But as far as capabilities, I’m looking for the following:

• Does it use response playbooks? I want a way to define custom scenarios or import my existing playbooks into the platform. That way, the platform is aware not just of best practices but also of my internal processes. In my own testing, I tried running it without any guidance from my playbooks, and the results weren't great. The investigation quality actually got worse over time.

• What's the right mix of guidance? I also tried adding my playbooks and letting the AI mainly follow that, and the results were good but a bit limited. We realized this would mean constantly updating our playbooks. The third approach, which got the best results, was to combine our internal playbooks with the platform's best practices. It understood our environment better and used all the log sources it needed to draw a conclusion. Plus, it took paths that were not in our playbooks but made sense in that specific context.

• Don't fall for the speed trap. These platforms mainly run API queries or search the SIEM. This means some investigations will only be as fast as your other tech. You have API rate limits and SIEM performance limitations. (Note: if your SIEM pricing is based on the number of queries you run, an AI will go a bit wild and run many more queries than a human would).

• Parallel vs. Sequential. It's also important to check if the platform runs actions in parallel or one after another. In security investigations, parallel actions don't always work because you often need to pivot based on what you find. Parallel makes sense for enrichment, but not for reactive threat hunting.

• Self-Verification. And one last point: does the AI verify and check itself once it completes an investigation? I've seen that when it does, it usually comes up with better outcomes.

• The Copilot. I can't forget the AI Copilot. At first, I thought it was mainly for starting a threat hunt, but I also want it available during an investigation. You should be able to pick up where the case was escalated and ask questions about specific steps or just continue the investigation yourself. It's important to make these interactions visible so all analysts can see what others have investigated. Good auditing and sharing options are key.

Taking Action: Remediation and Response

On this part, not many platforms are going all-in, mainly because response is harder to automate and it's something where you want more deterministic, predictable automation. From what I’ve seen, there are three types of capabilities:

1. Suggestions: They come up with response and remediation suggestions. This is the baseline I would expect. The key here is whether they understand your environment and suggest something that makes sense or just give you generic best-practice advice.

2. Basic Actions: These are the ones that have some simple response actions, like blocking an IP, suspecting a user session, resetting a password, doing an AI Bot interview with a user, alerting via IM, or quarantining a machine. I like this, as in some cases that’s all you need. It’s easy and convenient.

3. Native Automation: This is the third type, where platforms start to offer native automation capabilities like those in a SOAR platform, where you can build your own custom response workflows.

Closing the Loop: Lessons Learned

And the last part. Here we have a few capabilities to look for.

• Summaries & Reports: All platforms do alert and incident summaries; it’s like a core feature. But there are different types. Some allow you to create executive reports and give them a template for how it should look. You don't want to present reports to your management that are in a different style every time, so consistency is important. Others can do a summary of specific alerts you select and let you add artifacts to tailor it.

• The Feedback Loop: This is where things get interesting.

  • Some platforms give you no feedback mechanism. :)

  • Then you have the ones that offer basic suggestions, usually based on a summary report or dashboard.

  • Others come with suggestions on how to improve your detections. (This is my favorite,it’s what I want and I’ve even started to use it).

  • The ideal is a platform that gives you suggestions for both detection improvements and process improvements.

Final Thoughts

Evaluating an AI SOC platform isn't about finding a single killer feature. It's about understanding how all these different pieces, ingestion, knowledge, investigation, response, and feedback, work together. The goal isn't just to find a tool that closes alerts fast. It's to find a partner that can learn from your environment, adapt to your processes, and ultimately make your entire security program smarter. You need a system that thinks and learns with you, not just another black box.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

I’ve been going over a bunch of AI SOC implementations lately, and something hit me: control. It’s not just about having an “autonomous” system that investigates alerts. It’s about being able to see why it took a certain path, adjust the logic, and align it with your own environment. Without that, you end up with a black box that you can’t fully trust. 

This isn’t just theory. In almost every SOC I’ve worked with or advised, I’ve seen how fragile it becomes when you either (a) rely fully on vendor defaults, or (b) build out hundreds of breakable playbooks by hand. Both lead to pain in different ways.

This edition is sponsored by D3 Morpheus

Thinking AI SOC? Think Morpheus

Thinking AI SOC? Think Morpheus.
Morpheus delivers autonomy + control.
Built for Enterprise SOCs and top-tier MSSP/MDR.

▪️100% alert coverage
▪️ 95% alerts triaged in <2 mins.
▪️ Playbooks: autonomously built, user-adaptable 

24-7-365 autonomous investigations across any SOC stack, plus built-in case mgmt., IR flows, dashboards. 800+ integrations.

CISO/SOC/MSSP pros, we’d love to meet you.

Mapping the AI SOC Landscape

When I look at AI SOC platforms, I break them down into two ways.

First, on which stage of the incident lifecycle do they focus:

• Left - the data side: pipelines, log processing, detection engineering.

• Middle - enrichment, triage, and investigations.

• Right -response, remediation, and the feedback loop.

Some vendors specialize in one stage. Others stretch across multiple, but rarely all three.

Second, by implementation style:

• Plug-and-Play -quick to deploy, usually centered on the middle (triage, investigations). You connect data sources, and it starts producing outcomes with minimal setup.

• Build-and-Customize -more like next-gen security automation platforms. These let you create workflows from scratch, wire them to alerts, or even run “headless” background automations. They usually cover the middle and right, sometimes the left as well.

Over the last 2–3 years, most new AI SOC startups have landed in the middle with plug-and-play products. Security automation platforms lean toward build-and-customize.

Both approaches have tradeoffs, and understanding them is key.

The “Build-and-Customize” Trap

Build-and-customize platforms feel like classic real-time strategy games. You start with nothing and design everything yourself: the integrations, the escalation logic, the deterministic flows. You’re in charge.

That control is great, but it also means you own the complexity.

Here’s the trap: deterministic automation doesn’t scale well if you try to build one workflow for every single detection use case. In one SOC I worked with, the team had over 200 automations, each tied to a specific detection. The result? They needed more engineers than analysts just to keep workflows alive. Every time an API changed, a field was renamed, or a vendor shifted its schema, half the playbooks broke.

So while this model is powerful, it quickly becomes breakable. It works well if you’re building “macro” workflows (like log ingestion or enrichment pipelines). But if you try to encode every micro-decision into a deterministic playbook, you end up with a maintenance nightmare.

The “Plug-and-Play” Black Box

Plug-and-play AI SOCs are the opposite. Think of them as strategy RPGs: the world is pre-built, the rules are set, and you just play the role you’re given.

The upside is obvious: fast time-to-value. You connect your SIEM or log sources, and suddenly the platform is triaging, clustering, or even investigating alerts for you. For many orgs that are short-staffed, that’s appealing.

The downside is just as obvious: no visibility into how the logic works, and no way to adjust it.

• What if the platform suppresses something you consider critical?

• What if its default enrichment path doesn’t fit your environment?

• What if you just want to add one custom validation step before remediation?

In most cases, you can’t. You’re locked into the vendor’s logic. And when your team doesn’t understand or can’t shape the process, trust in the tool drops fast.

Where Automation Already Works Well

Before talking about hybrids, let’s ground this in where automation already delivers value. From my own experience, the biggest wins are on the left side of the lifecycle.

Take log ingestion. Many SaaS tools don’t provide streaming log integrations. You either need to pull data via API or pay for a third-party connector. I’ve built automations that:

• Fetch audit logs on a schedule,

• Parse and normalize them,

• Push them into S3 or another bucket,

• Then feed them to a SIEM in the format it expects.

The benefit is twofold: the SIEM gets data in a clean, expected schema, and you offload parsing/storage from the SIEM, which saves costs.

In detection engineering, I’ve built workflows that enrich threat intel feeds, extract TTPs, and generate hypotheses for new detections. For example, if intel shows a threat actor shifting to a new initial access vector, the automation surfaces that and suggests where detection coverage may be missing.

There are also operational automations:

• Creating backlog tickets automatically when a false positive is reported.

• Triggering attack simulations when new detections are deployed.

• Sending requests to red teams to validate coverage through adversary emulation.

These aren’t glamorous, but they save huge amounts of manual effort.

The Messy Middle

The middle of the SOC lifecycle isn’t just “messy;” it’s where the real detective work happens. You’ve pulled in enrichment and gathered context, and now you need to connect the dots into a coherent incident story.

This is harder than it looks. Evidence isn’t uniform. Sometimes you’re pulling infrastructure data and need to cross-check with change requests or asset management logs. Sometimes it’s endpoint behavior, where there is no clear baseline, users run all sorts of random processes on their machines, and you need to know what’s “normal” in this business context. Other times, you’re correlating identity data, asking whether a login pattern aligns with known behavior for that role, that region, or that application.

In practice, analysts spend much of their time bouncing between SIEM queries, log sources, and business systems, running searches and pulling fragments of context. The challenge is less about gathering raw data and more about stitching it into a narrative that makes sense. That’s why the middle ground has historically been so fragile for automation; you can’t just script it once and call it done. The process changes with every environment, every investigation, every new clue.

This is where AI could be transformative, if it helps analysts assemble evidence and propose connections, but still shows why it drew those links. Without that transparency, it’s just guessing in the dark.

A Hybrid Model: Autonomy with Guardrails

So instead of being stuck with a black-box tool or a maintenance nightmare, imagine something in the middle. A hybrid model is all about getting the best of both worlds: the speed and scale of AI automation, with the control and flexibility your team actually needs.

Here’s how I see it working: Instead of coding every playbook by hand, you just drop an alert into the AI SOC platform. From there, it generates a full investigation workflow on the fly.

But here's the key part, it's not a black box. You can see all the steps it plans to take, and you can fine-tune them as needed. You could even upload one of your existing runbooks from Confluence, and the platform would use it as a template to shape its automation logic.

I like to think of it with a gaming analogy.

• The "City-Builder" Phase: This is the build-and-customize part where you lay down the rules. You set up the integrations, define your critical assets, and build deterministic guardrails, like "for any action on a domain controller, you must get human approval".

• The "RPG" Phase: This is where the autonomous AI agents operate within the world you just built. They can investigate alerts, enrich data, and even suggest remediation, but they always have to follow the rules and stay on the roads you created.

This approach combines the strengths of both copilots and fully autonomous agents. Key capabilities usually include:

Balanced Autonomy: The system handles the routine, high-volume stuff on its own but knows when to stop and escalate tricky or high-impact decisions to a human analyst.

Flexible Exploration: Your analysts aren't locked into a rigid workflow. They can pivot from the AI's automated findings and start asking their own questions in an interactive chat, letting them dig deeper whenever they need to.

Customizable Logic: The AI's workflows can be tailored to fit your SOC’s specific needs, giving you a good balance between automated consistency and the flexibility to handle unique threats.

Ultimately, this balance, autonomy with guardrails, is what will make AI SOCs something we can actually rely on. It gives you a system that can handle the massive scale of alerts while making sure a human is still in the driver's seat for the decisions that really matter.

Final Thoughts

Look, at the end of the day, there's no magic answer here. It’s super important to figure out what fits your own environment. Both the plug-and-play and the build-it-yourself platforms have their place.

If you're short on people and just need something running fast, a plug-and-play tool is tempting. But you're stuck in their world, using their logic, and it's basically a black box. On the other hand, if you have a big engineering team, maybe building everything from scratch sounds good. But I've seen how that turns into a maintenance nightmare that costs a fortune to keep running.

This is where the hybrid model has some serious advantages. It's way faster to deploy than trying to build everything custom from the ground up, since the core AI investigation logic is already there. The cost of maintenance is also a lot lower because you’re not trying to keep hundreds of automation playbooks alive. And you still get the flexibility to tweak the workflows and make sure the system operates in a way you can actually trust. You get the speed of AI without having to give up all the control.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Audio Version

I’ll take some responsibility here. Maybe I started this movement, maybe not, but I’ve definitely been pushing it for a while. So now I’ll give it a proper name and see if I get some credit for it. In the end, it’s not about who said it first. It’s about sharing knowledge and helping the community make sense of what’s happening.

It's been a couple of years now since AI really started hitting the cyber field, especially in Security Operations. We started with AI copilots, then the focus moved to the sweet spot, investigations, and now we're seeing a push to "shift left" and "shift right."

And if you’re wondering what I mean by shifting left or right in this context, let’s clear that up. Unlike the messy attempts of “shift left” in SDLC, here it’s simple: shifting left or right means asking which stage of the IR cycle you are applying AI to. For me, that’s the best way to evaluate, identify, and understand AI’s true capabilities for SecOps.

So let’s give this framework a name: The SecOps AI Shift Map.

This edition is sponsored by Exaforce

From Zero to AI-Driven SOC

Exaforce is a breakthrough AI SOC platform that infuses AI into every stage of your SOC lifecycle, across detection, triage, investigation, and response. We help you reduce your manual effort and improve your outcomes, while driving down costs. To learn how Exaforce is helping organizations like yours, book a call with us or join the upcoming webinar.

Shift Left: The Land of Detections and Data

If you break down the IR cycle into three stages, left, middle, and right, you can start mapping where AI is actually being used. On the left side, we have data ingestion, log processing, and detection engineering. Think of it as the foundation: we set up controls like EDR, network protection, IAM, cloud guardrails, email security, the usual stack. 

Then comes logging. You centralize everything in a SIEM, prioritize sources, and slowly build toward the impossible dream of “100% visibility.” 

After that comes detection engineering: choosing the right log sources, building threat profiles, and designing detections based on expected TTPs. That’s the left side of the house, and it ends with a detection firing.

The Middle: The Investigation Sweet Spot

The middle side is where things get interesting, and honestly, where most of the industry is today. Once a detection fires, you need procedures. Call them SOPs, runbooks, or playbooks, they all serve the same purpose: guiding what you need to look at. Here comes enrichment, and I know there are strong opinions on whether this should sit at the detection layer or the investigation layer. In reality, it depends. Some enrichments definitely make detections stronger, but many, like CTI lookups, alert similarity, correlation, or even tribal knowledge, make more sense during investigation.

Once enriched, the investigation starts. I like to map this stage to the 5Ws: Who, What, When, Where, and Why. Answering those questions leads you to a conclusion, a verdict.

We have a few types of verdicts:

• True Positive: This is malicious; we have a compromise or an issue that needs to be fixed.

• Benign (False Positive): This alert shouldn't have triggered, and the detection needs to be fine-tuned.

• Benign (True Positive): The alert triggered as expected, and the activity was confirmed to be normal. (Technically, some folks call this a "Benign True Positive" since the alert did fire correctly, but the point is the same: the activity was expected and not a threat.)

Note that we are measuring metrics on alerts here, not incidents. Incidents are a different bucket. They can start directly in the investigation phase without an alert, and that’s where we might find a False Negative, no alert was generated, but someone (an employee or an external party) identified a compromise.

Shift Right: Remediation, Recovery, and Broken Feedback Loops

And now for the right side of the house. This is where remediation, recovery, and lessons learned happen. It’s also the hardest part to automate or apply a lot of AI to, especially remediation.

You’ll hear many people say this is where SOAR failed. I’d say no, it didn’t, we failed here because our processes are a mess. True automation hits a wall when it runs into a lack of API coverage for critical internal tools, undocumented tribal knowledge, a culture of risk aversion, and of course, the dreaded change management process. Remediation isn't always straightforward. The larger and more regulated the organization, the more you run into these roadblocks. You can’t just go wild and do what you want. No one wants to give a service account to a SOAR or AI SOC solution that has full write access. Even if some actions are pre-approved, the risk is just too high.

The idea is that you don't have many compromises, so you shouldn't need to run these actions often. This beats the purpose of building automations that run only a few times a year. (And if you're an org that needs to run these actions often, I think you should go back to the left side and fix your issues there).

Recovery is a bit easier to automate, and this is where IT and DevOps automation usually comes into play. And lessons learned? This is where AI shines. It’s great at writing a summary, getting you an executive report, and formatting it nicely. The part we need to improve is how we feed this information all the way back to the left side to constantly get better. We’re not doing a great job there. Where we fall short is feeding those lessons back to the left side to improve detections and logging. And let’s be honest: for many vendors, fewer problems don’t translate into revenue, so the feedback loop rarely gets prioritized.

So, Why Did Everyone Start in the Middle? (The Cake Analogy)

When AI for SecOps first showed up, vendors went straight for the middle, the investigation. It’s like getting a fancy cake. You ignore the heavy fondant on the outside (the broken detections) and go for the sweet, sugary center. It gives you a quick sugar rush (hundreds of alerts closed fast!), and you feel great.

(Weird analogy, I know).

But you're not fixing the real problems. The fondant still tastes bad, and the sugar crash is coming. The "lessons learned" are that you ate too much cake, but a week later, you forget and just remember the sweet part. You want that good feeling of eating the sweet middle part (all those 100 alerts closed fast).

So yeah, a long way of saying it started in the middle because it was the easiest part. It's the main area where we don't need as many deterministic things, and we can easily use GenAI to do analysis and come up with a verdict.

Beyond the Investigation: Fixing the Whole Problem

But now we're realizing that just doing the investigation isn't enough. What about our broken or incomplete detections? What about the log sources that are missing key logs? What about the simple response actions?

That's what we need to actually fix the problem. I want to get recommendations on which detections should be fixed and which log sources should be improved. I want my AI to be able to reach out to a user and ask for more information if needed, and then quarantine their machine and reset their password. 

Simple, right?

Sad story is, not many are doing it yet.

Final Thoughts

The SecOps AI Shift Map is a way to frame where AI is being applied across the IR cycle. It helps us evaluate vendors, set expectations, and identify gaps.

We started in the middle, because it was easy and satisfying. But the real progress ,the real SOC of the future ,is in shifting left and right. That’s how we’ll build AI SOCs that don’t just close alerts faster, but actually make security operations stronger end to end.

Just before I jump to the vendor spotlight, I wanted to reassure you, the audience: all the vendors I mention here are ones I've had a demo with to understand their capabilities. It’s not just for the sake of having a vendor highlight; it's something I've seen and evaluated, and I'm giving you my insight on what they're good at.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Memory makes GenAI useful in the SOC and also makes it opinionated. That opinion can drift. Some stacks become optimistic and default to benign. Others become pessimistic and default to malicious. Both are forms of bias. You can control this with explicit objectives, memory hygiene, calibration, and drift monitoring. Citations and a practical checklist are below.

Table of Contents

• The observation

• Where the bias actually comes from

• Monitoring Drift in AI SOCs

  • Population Stability Index (PSI)

  • ADWIN (Adaptive Windowing)

  • Why this matters in practice

• Why AI SOC Drift Is Different From Traditional Mod …

• How to Control Optimism vs Pessimism on Purpose

  • 1. State your loss function

  • 2. Separate facts from verdicts in memory

  • 3. Enforce memory hygiene

  • 4. Use structured, two-pass reasoning

  • 5. Calibrate regularly

  • 6. Monitor drift continuously

  • 7. Test with counterfactuals and red teams

• What to ask your vendor

• What we know from the literature and industry

• References

• Closing view

The observation

In a clean world, AI investigations end in one of three outcomes: benign, suspicious, or malicious. In the field, once you add memory, the model starts anchoring to prior cases and narratives. Over time, I see two failure modes:

• Optimistic drift: the AI looks for evidence that activity is benign and closes too fast.

• Pessimistic drift: the AI assumes breach and marks too many items as malicious, flooding triage.

Both are biased patterns reinforced by memory. This is not “AI gone wrong.” It is a predictable effect of feedback, sampling, and incentives. Anchoring, confirmation, and availability biases show up in ML pipelines just as they do in humans.

My take: I prefer a pessimistic default in security. Assume compromise and plan for worst case, then reduce noise with controls rather than miss a critical event.

Do you enjoy CyberSec Automation Blog Content?

Nominate for SANS Difference Makers Awards

The SANS Difference Maker Awards are open for nominations, and I’d love your support. I’m applying in the Media Creator of the Year category for my work on CyberSec Automation. If you like the content I’ve been sharing, it would mean a lot if you could take a minute to nominate.

Where the bias actually comes from

• Anchoring via memory

  Retrieval of previous conclusions, tickets, and notes can anchor the model’s current reasoning. If memory stores verdicts, not only facts, you increase confirmation bias. NIST and ACM both call out bias as socio-technical, not just model-level.

• Label/feedback loops

  If analysts reward “fast closes,” your reinforcement signal pushes optimism. If you reward “catch anything suspicious,” you push pessimism.

• Dataset shift and model drift

  Your traffic, tools, and attacker mix change. That is covariate shift and concept drift. You must detect and respond to it. Practical detectors include PSI for distribution shift and streaming detectors like ADWIN.

• Calibration decay

  Confidence scores stop matching reality as the environment changes. Track Expected Calibration Error and similar metrics to keep “probability of malicious” honest.

• Domain specifics of cyber

  In cybersecurity, bias does not only waste time. It can either hide attacks (optimistic) or burn out analysts and suppress signal (pessimistic). Industry pieces echo this tension in practice.

Monitoring Drift in AI SOCs

Two practical techniques you’ll hear about in ML Ops — and that are directly useful in AI SOC — are PSI and ADWIN. Both are ways of spotting when your model has started to “see the world differently” than it did at training or deployment time.

Population Stability Index (PSI)

• PSI is a simple way to measure whether the distribution of features (inputs the model uses) has shifted.

• For example: imagine your model relies heavily on login geolocation or file hash rarity. If the frequency distribution of those features changes a lot compared to your baseline (say, suddenly 30% of logins are from a new region), your model is now making predictions on data it hasn’t really been trained for.

• We typically set thresholds:

  • PSI < 0.1 → stable (no real change)

  • 0.1–0.2 → moderate shift (keep an eye on it)

  • >0.2 → warning level, drift may be affecting results

  • >0.25 → action required (retrain or re-evaluate)

• In the SOC, you’d use PSI on high-value features like source IP reputation scores, authentication method, or endpoint process ancestry — the signals that drive most of your verdicts.

ADWIN (Adaptive Windowing)

• ADWIN is a streaming drift detector. It looks at incoming telemetry in real time and detects if the statistical properties of the data have changed.

• Think of it as a moving window: if the recent data looks very different from the older data, ADWIN flags drift.

• Example in a SOC:

  • You’re monitoring failed login attempts per user per hour. Normally the distribution is steady. Suddenly, in the last hour, the rate jumps in a way that doesn’t fit past behavior. ADWIN detects this as a distribution change — signaling that the model’s prior assumptions may no longer hold.

• ADWIN is valuable when your SOC ingests continuous, fast-changing data (auth logs, endpoint events, netflow).

Why this matters in practice

• PSI tells you when your AI is drifting because the population of data has shifted.

• ADWIN tells you when your AI is drifting because the stream of data is behaving differently in real time.

Both should trigger drift alerts in your SOC platform. When thresholds are hit, you either retrain the model, adjust decision thresholds, or at least shadow-test the model to confirm it’s still making reliable calls.

Why AI SOC Drift Is Different From Traditional Model Drift

There’s something specific to cybersecurity worth calling out.

When you roll out an AI SOC platform, you’ll either:

• Feed it historical data (if the platform supports it), or

• Let it start fresh, learning in shadow mode from day one of deployment.

In the traditional SOC workflow, when a human analyst processes an alert, the output is often a one-liner:

• “False positive because xyz.”

• “Normal activity, change request attached.”

That limited context doesn’t feed the model much bias. It tells the system how the case ended but doesn’t provide a lot of rich narrative that can anchor future decisions.

But with AI handling investigations, the story changes. Instead of one-liners, you get one, two, sometimes three paragraphs of reasoning explaining why an alert was closed as benign, suspicious, or malicious.

And here’s the kicker: most AI SOC platforms are built so that analysts approve or deny those AI conclusions. That means the model isn’t just learning the verdict — it’s learning from the entire summary and narrative it generated.

The result?

• The more alerts you approve with “benign” narratives, the more the model drifts optimistic.

• The more you approve “malicious” narratives, the more it drifts pessimistic.

• And beyond optimism/pessimism, the model starts encoding other forms of bias hidden in the text — anchoring to particular arguments, data sources, or analyst preferences.

This makes AI SOC drift qualitatively different from classical ML drift. You’re not just feeding it labels. You’re feeding it reinforced narratives — which are far more context-rich and far more prone to anchoring.

How to Control Optimism vs Pessimism on Purpose

1. State your loss function

Every SOC has to decide which mistake is more costly:

• False Negative (FN) → Missing an intrusion.

• False Positive (FP) → Investigating noise.

In most environments, FN > FP — a breach is worse than wasted cycles. But not all teams weigh it the same:

• A resource-constrained SOC may set stricter limits on FPs to avoid burnout.

• A high-risk sector (finance, healthcare) will tolerate more noise to minimize missed threats.

The key is to make this explicit. Don’t let the model’s default bias define your risk appetite. Encode the loss function into:

• Thresholds: e.g., “AI must be 85% confident to close as benign, but only 60% confident to escalate as malicious.”

• Routing rules: e.g., suspicious verdicts always go to Tier 1, but any high-risk suspicious (based on threat intel correlation) goes straight to IR.

Treat this as SOC policy, not a hidden “prompt hack.”

2. Separate facts from verdicts in memory

One of the biggest sources of anchoring bias is how memory retrieves past cases.

• If the AI sees a verdict like “benign – normal user behavior” in memory, it may bias its current conclusion toward benign.

• If memory only retrieves facts (e.g., “User X logged in from IP Y at 03:00, MFA success”), the AI can evaluate evidence without inheriting the past verdict.

Practical controls:

• Store artifacts, signals, and context → log snippets, process trees, enrichment.

• Down-weight final labels → allow retrieval of verdicts, but apply less importance than raw evidence.

• Force “evidence-first” prompts → e.g., “Summarize the facts before giving a conclusion.”

This keeps memory as a knowledge base, not a verdict repeater.

3. Enforce memory hygiene

Think of memory like a SIEM data lake — garbage in, garbage out. Without hygiene, bias compounds.

• TTL (Time to Live): Don’t let outdated conclusions anchor current cases. E.g., verdicts expire after 30 days unless reaffirmed.

• Source tags & provenance: Every memory chunk should record its origin — log type, analyst name, AI agent version. This makes retrieval explainable.

• Retrieval filters: Prefer multi-source evidence. For example, if two independent log sources (e.g., auth + EDR) align, rank that memory higher than a single noisy source.

This reduces toxic bias accumulation and prevents “zombie verdicts” from skewing current reasoning.

4. Use structured, two-pass reasoning

Humans avoid confirmation bias by debating, and AI needs the same. A two-pass system creates an internal “red team.”

• Pass A: Build a case file of observations and hypotheses. E.g., “Unusual PowerShell execution observed, correlated with new registry keys.”

• Pass B: Adversarial review. AI (or a secondary agent) argues the opposite verdict. E.g., “This could also be normal IT admin activity — prior change tickets show similar actions.”

The final verdict must resolve both arguments. This mirrors human analyst workflows (peer review, escalation) and makes conclusions more balanced.

5. Calibrate regularly

Raw confidence scores are almost always misleading. Calibration ensures “80% confidence” really means “8 out of 10 times correct.”

• Metrics:

  • ECE (Expected Calibration Error): measures mismatch between predicted vs actual accuracy.

  • Brier score: penalizes both overconfidence and underconfidence.

• Operationalization:

  • Re-tune thresholds if calibration drifts.

  • Publish a monthly calibration report to SOC leadership showing whether the AI’s confidence is still trustworthy.

Well-calibrated AI allows you to set rational escalation policies instead of “gut feel” thresholds.

6. Monitor drift continuously

Data never stays static. Model drift is inevitable. The only question is whether you catch it.

• PSI (Population Stability Index):

  • Compares historical feature distributions vs live data.

  • E.g., login location mix changes drastically.

  • Thresholds: >0.2 = warning, >0.25 = action.

• ADWIN (Adaptive Windowing):

  • Monitors real-time streams for sudden changes.

  • E.g., spike in failed logins per user compared to historical patterns.

• Response:

  • Trigger shadow evaluation, partial retraining, or force suspicious verdicts when drift is detected.

  • Automate drift alerts into the SOC dashboard (same way we alert on log ingestion failures).

7. Test with counterfactuals and red teams

SOC AI should be tested like detections: continuously and adversarially.

• Counterfactuals: Hold out known attack scenarios and near-miss benigns. Evaluate AI weekly against this set.

• Rotation: Refresh test data monthly to avoid overfitting.

• Red teaming: Actively simulate edge cases — “what if MFA fails but user behavior is normal?” — to stress-test reasoning.

NIST AI RMF highlights this: continuous evaluation tied to business harm, not just technical accuracy.

What to ask your vendor

• How do you prevent anchoring from prior verdicts when using memory or case history?

• Do you provide calibration reports and can we export ECE or similar?

• What drift detectors are built-in (PSI, ADWIN, custom) and how are alerts surfaced?

• Can we tune the loss function or cost ratios for FN vs FP by use case?

• Do you support two-agent review or adversarial reasoning before final verdict?

• How do you version prompts, memories, and model configs so we can audit changes?

• Show us your approach aligned with NIST AI RMF controls for measurement and monitoring.

What we know from the literature and industry

• Bias can enter at data, model, and deployment stages. You must treat this as a socio-technical problem, not just tuning a model.

• Cybersecurity is already seeing bias outcomes that either hide threats or inflate noise. Leaders are concerned and are asking for measurable controls.

• Drift is normal in live systems. Use PSI for distribution monitoring and ADWIN for streaming change detection. Build playbooks that trigger re-evaluation when these fire.

Confidence must be calibrated if you expect analysts to trust scores. Track ECE and retrain or re-threshold when it degrades.

References

• ACM: Biases in AI Systems

  https://cacm.acm.org/practice/biases-in-ai-systems/

• Interface Media: Exploring the Impact of AI Bias on Cybersecurity

  https://interface.media/blog/2024/12/24/exploring-the-impact-of-ai-bias-on-cybersecurity/

• Chief Executive: Why AI Bias Is a Growing Cybersecurity Concern

  https://chiefexecutive.net/why-ai-bias-is-a-growing-cybersecurity-concern/

• NIST AI Risk Management Framework (AI RMF 1.0)

  https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf

• Concept Drift Detection in Data Streams (ADWIN, Gama et al.)

  https://homes.di.unimi.it/~cesabian/Pubblicazioni/mlj10.pdf

• Measuring Calibration in Deep Learning (Guo et al., 2017)

  https://arxiv.org/abs/1706.04599

• Population Stability Index (PSI) for Monitoring Model Drift (Practical Guide)

  https://docs.snowflake.com/en/user-guide/mlops-psi

• NIST: Bias in Artificial Intelligence

  https://www.nist.gov/news-events/news/2022/03/nist-study-evaluates-tools-mitigate-bias-artificial-intelligence

Closing view

I still prefer a pessimistic default in security. Assume breach. Pay the cost of extra triage while you mature memory, calibration, and drift controls. You can dial back noise with evidence requirements and better calibration. You cannot easily recover from a missed intrusion.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Here's another edition of the CyberSec Automation blog, and you're probably noticing it looks a little different. I recently launched the CyberSec Automation Interview Series / Podcast, and I wanted to share some of the recent episodes with you.

In these sessions, I chat with founders of cybersecurity companies, especially in the SecOps space, and other practitioners. The idea is to have a casual, no-BS conversation about what's really happening in security automation. Events are streamed live on LinkedIn, and I've also created a YouTube channel where you can catch up on past episodes. Go ahead and subscribe if you want to get notified about new ones.

The episodes are usually tied to a blog post, making it easier to follow the topic. Some are vendor-agnostic, while in others, founders explain how they're tackling a specific problem with their solution.

This monthly edition of the blog will be all about the podcasts I host and the ones where I'm a guest.

Since this is the first one, here are the episodes and the related blogs.

When Does It Make Sense to Automate? When Does AI SOC Actually Work?

Join us for the next episode of the CyberSec Automation Interview Series!

In this episode, I sit down with security pros Andrei Cotaie and Cristian Miron to tackle a big question in modern SOCs: when does it make sense to automate, and when should we trust AI in the SOC?

We'll talk about the three main stages of the SOC workflow: detection and log ingestion, investigation, and response. We'll get into where automation provides the most bang for your buck, where AI SOC is starting to make a real impact, and where it’s still too early to rely on it.

It's just a laid-back chat with real-world examples of how we're implementing and using AI SOC and automation.

You can check out the upcoming event here: https://www.linkedin.com/events/whendoesitmakesensetoautomate-w7360612707361841152/

And the related blog post is here:

Why SOC Analysts Ignore Your Playbooks

Uncover why SOC analysts resist traditional playbooks and learn how to create more effective, actionable security automation strategies that actually work.

Smashing the Myth of the Single Pane of Glass

I sat down with Senad Aruc, CEO of Imperum, to deconstruct the "single pane of glass" architecture.

We got pretty technical, talking about why simple LLM wrappers don't work without native Threat Detection, Investigation, and Response (TDIR), the future of SIEM, how to deal with API limitations, and the kind of stack you need for a real Autonomous SOC.

Watch the full episode here:

And here's the blog post that goes with it:

Stop Chasing the Single Pane of Glass

Debunk the "single pane of glass" myth in cybersecurity: Explore why chasing unified platforms falls short and discover smarter approaches to security monitoring and response.

Building AI for SOCs That Analysts Don’t Hate

In this episode, Tom Findling from Conifers.ai joins me to talk about what it takes to build an AI-driven SOC platform that analysts actually want to use.

We dive into the key features every AI SOC platform should have, best practices for implementation, and how to set realistic expectations from the get-go. No hype, just practical advice.

Check out the conversation here:

The related blog is:

Automate Smarter, Not Louder: Using Interactive AI Feedback Loops

Revolutionize SecOps with AI-driven feedback loops: Learn how smart automation transforms threat intelligence, detection, and response strategies effectively.

Where I Was a Guest

I also had the chance to be a guest on a couple of other podcasts.

EP 6. From Data Chaos to Detection Engineering: How to Automate What Really Matters in the SOC

I joined Balázs Scheidler on the Data Strikes Back podcast to talk about a topic that's close to my heart: your SOC isn’t failing because of bad detections; it's failing because your data is a mess.

We talked about:

• How to use automation for more than just incident response, like in log management and data pipelines.

• Why "detection as code" is a waste of time without good schema discipline.

• How to handle massive amounts of legacy syslog data (we're talking 50TB/day) without blowing your SIEM budget.

• When it makes sense to standardize, transform, or just automate everything.

You can listen to it here: Data Strikes Back on Spotify 

Defender Fridays by LimaCharlie

I was also invited to join an episode of Defender Fridays by LimaCharlie. It was a great conversation about the current state of security automation and where things are headed.

You can watch that here: Defender Fridays

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Audible version of the blog

The SOC playbook. It's been the backbone of security operations since cybersecurity became a thing. I started my career in this field 15 years ago, and honestly, not much has changed about the concept. A playbook is like a recipe; it's a simple idea that just works, and there's not a whole lot you can do to change the core concept. What has changed is how we interact with it.

Initially, it was just humans following the steps. Then, automation came along, and we started using SOAR platforms to automate bits and pieces of the playbook. Now, we have AI, and surprise, we're using it to write playbooks or have it follow the steps for us.

The whole point of a playbook was to solve the "tribal knowledge" problem. It's simple: document your processes so everyone is on the same page.

But here’s the sad story: they are broken, and they have been since we started using them.

AI should think like your best SOC analyst, not someone else’s. Turn your team’s best instincts into AI-driven workflows without any integrations with Legion

Why Your "Golden Standard" is Gathering Dust

Don't get me wrong, there are cases where playbooks work perfectly. In highly regulated environments, for instance, they have to be followed to the letter (which, in my view, is a fast track to analyst burnout). But in most SOCs? Not so much.

From what I’ve seen, someone writes them, and they’re great for new joiners to understand how a specific organisation works. But are they followed consistently? Nope. Unless you have a deterministic automation platform that forces those exact steps, analysts tend to go off-script.

Experienced analysts, especially those who have been in the SecOps space for a while, usually go by instinct. If I asked you to pull the alerts from the past year, analyse the summaries, and check if the playbook steps were followed, you'd find a huge percentage are outliers.

And you might think, "who cares, as long as the job gets done?" Yeah, that's true. But I think this is the one loose brick in the SecOps tower that causes the butterfly effect, making everything else seem broken afterward. Now you're wondering, "ok, this seems off."

Let's break it down.

• Why do we have too many unhandled alerts?
  Because your feedback loop is broken. You don't have time for fine-tuning detections. And why is the feedback loop broken? Because that playbook step that says, "every false positive needs to be sent back to detection engineering," is being ignored by everyone.

• Why does your analyst retention rate suck?
  Because they handle the same boring stuff over and over. They see the same alert, and their brain runs the same internal playbook instinctively. We have tools to automate this, but what about the edge cases? Do they follow a playbook for those, or do they just close another ticket as "False Positive" or "Normal Activity"? Probably the latter. This happens because they either don't follow the playbook, or the playbook hasn't been updated in so long it's irrelevant.

So yeah, I can go on, but I think many of the biggest problems in the SOC trace back to one thing: a broken playbook equals a broken process. It’s funny because of the classic PPT (People, Process, Technology) model, the process is almost always the thing that’s broken.

• You have the people and tech, but your process is a mess (most common).

• You have people and processes, but your tech is garbage (happens sometimes).

• You have tech and processes but no people (I haven't heard of this one, but who knows, maybe it's possible).

So, How Do We Fix This Mess?

Ok, enough ranting. You came to this blog to hear about solutions. Can we fix it?

I think yes. I was able to build and implement a framework that helps fix this. The main idea is to implement automation as much as possible, but you have to do it in a specific way.

Start on the left side: Data Ingestion and Detection Engineering. This part is closely tied to threat hunting and threat intel, where your hypotheses and threat profiles begin. You know the basics: understand your environment, know what you need to defend, and know who you're defending it from. Build that basic structure of Assets and Identities—the key to good detections is having this context.

Once you have a new detection (after you evaluate, develop, and test it), you need to figure out what the playbook will be for it. This is the first breaking point for most teams. A detection without a playbook is one of the main causes of bad processes.

Here's a classic example: a new, scary threat emerges. You rush to develop a detection and throw it into production. The first week, everyone knows what it’s about. The first month, maybe. But then the alert goes quiet for a while. The analyst who wrote the detection leaves the company. A year later, you have this shadow detection that no one understands. Now it’s broken, spewing false positives like crazy, and the analysts are just closing them on autopilot.

So yes, stop and build the damn playbook. It's easier than ever. You can throw the alert context into a GenAI tool, and it will spit out a decent draft in seconds. You can even get fancy and embed this step into your detection engineering automation process.

Once you have the playbook, you have the foundation to explore which steps can be automated. Simple, right? Even if it's just one or two steps, it’s a win. That’s how you start. Eventually, you can get to full end-to-end automations. You can even script this whole thing (I’m working on an open-source project for this, but it’s going slow, so stay tuned... subscribe to the blog, because LinkedIn might not show you my posts).

What if We Flipped the Model?

Now imagine this. What if a tool took a different approach? What if you just let your analysts do their job, letting them go wild and investigate alerts however they see fit? And what if this tool just monitored their investigation in the background and built the playbook by itself? Once it's done, the analyst could review it and turn it into an automation.

You might be thinking, "Filip, you just described an RPA solution." Yes, that's true. Sadly, no RPA solution has ever really worked for cybersecurity. The tech wasn't there, and the cyber environment is just too dynamic and unpredictable.

But now we have AI and Agents. We can add an undeterministic approach to these automations when needed. This approach is worth exploring, and there are vendors already working on it (check out the vendor spotlight section).

Final Thoughts: Stop Patching, Start Refactoring Your SOC

So, what's the big takeaway here? For years, we've been trying to patch the SOC. We throw more tools at it, we hire more people, we buy more threat intel feeds. It feels like we're just adding more features to a pile of spaghetti code. The real problem is in the core logic—the playbooks.

Those analysts going by "instinct"? They're not breaking the rules. They've just found a better, undocumented way to get the job done. They've built a more efficient process in their heads because the official one is slow, clunky, or just plain wrong. The issue is that this knowledge walks out the door when they do.

This is where that new approach I mentioned gets really interesting. Instead of forcing analysts to follow a rigid script, what if we had a system that could watch them work, learn their shortcuts, and turn that "instinct" into a documented, automated, and up-to-date playbook? This isn't about replacing the analyst. It's about cloning their expertise. It’s about building a process that adapts, instead of one that just gets old.

At the end of the day, a broken playbook is a broken process. And a broken process means your people and your expensive tech are never going to work as well as they should. So maybe, just maybe, before you buy the next shiny AI tool that promises to solve all your problems, take a hard look at your playbooks. That's the bug that's been causing your whole system to crash.

Vendor Spotlight: Legion Security

If AI can't learn how you work, you'll never trust it to work for you. Legion introduces a browser-based AI SOC companion that transforms your team’s expertise into scalable automation, eliminating the need for APIs, built-in playbooks, or integrations. 

Using a lightweight browser extension and AI vision models, Legion observes how your analysts work, capturing their decision-making processes, learning investigation patterns, and then automating them at your pace. It helps optimize workflows, operates 24/7 using your existing tools, and supports both autonomous and human-in-the-loop response. 

Designed to integrate instantly with any browser-accessible system, whether commercial platforms like SIEMs or homegrown tools, Legion’s AI SOC Analyst thinks like your team and reduces operational load without adding headcount. 

Turn your team’s best instincts into AI-driven workflows with Legion.

Legion Security | The Browser-Native AI SOC Analyst

Legion Security is the first browser-native AI SOC Analyst that learns your team’s actual workflows and scales that knowledge across your organization.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

Since the times I started my career in cybersec, the "single pane of glass" has been one of those holy grails. You know the dream, the imaginable heaven for cyber where we have one platform that has all the alerts, all the context, and automation for the response. Everything is just a click away.

Picture it: You put on your VR glasses and swipe over investigation cards. Next to you is sitting our AI robot that does all the heavy lifting, and we just push the big red button to say "malicious, crush it, burn it, later we will recover it." Okay, okay, I went too far with the imagination here. But yes, you get it. We all want a single pane of glass, and many vendors are trying to sell you one.

But if you ask yourself, what actually is a single pane of glass? How does it look? What does it do?

When I was at Adobe leading their Threat Intel and Threat Hunting programs, the CISO back then was Brad Arkin. He would ask me, "Filip, if I give you an unlimited budget, what would you do with it? What tool would you spend it on and why?" It was a good one. And I felt like I never had enough staff on my wish list. Why did I think that? Because I didn't know how to build the platform myself.

So now, it made me think... if I can build one now, how would it look?

And to be honest, I think there is no one solution that fits all. In my view, you need a solution that you can build on top of. Think of it like Legos, not a finished IKEA table. You need to build your own thing.

This edition is sponsored by Imperum

Your SOC. Supercharged by AI

Imperum is the only AI-driven, autonomous SecOps platform that unifies detection, investigation, and response – cutting through alert fatigue, manual overhead, and integration barriers. As the only connector-agnostic solution on the market, Imperum delivers hyperautomation, built-in intelligence, and full-spectrum coverage — turning your stack into a true security powerhouse.

Heading to Black Hat? Swing by Booth #6218 or book a meeting to see what happens when AI goes all in

What My "Single Pane of Glass" Would Track

So if I'm building this platform, what do I actually want to see and do? For me, it boils down to a few key things.

1.Assets: Machines and Identities

First, I want to track my assets. And really, there are only two types of assets I care about: machines and identities.

It helps to think of it like a classic fortress, you know? A bit old school, but it works.

• The fortress is your whole organization.

• Machines are the houses and buildings inside the walls. They are the static things, the structures. Your servers, laptops, containers.

• Identities are the people, the army, the livestock. They are the living things that move around and use the buildings. Your employees, service accounts, admins. They are the ones doing things.

• And the Network? That’s just the roads connecting everything.

When you think like this, it gets simple. Everything bad that happens in your network happens to one of these two things. If you start here, everything else gets easier. Tagging every event with an asset_id and a user_id is the magic key.

2.Your Data: A Fast Brain and a Perfect Memory

Let's be clear: your SIEM is the brain of your SOC. It’s where the magic happens—correlation, alerting, and hunting. But what happens when you try to force-feed it every single raw log from every tool you own?

It gets slow. It gets expensive. And your analysts spend more time waiting for queries to finish than actually investigating threats.

This is where a security data lake comes in. Think of it as a massive, cheap storage unit for all your security data—the good, the bad, and the ugly. You keep everything, but you don't force your SIEM to chew on it all at once.

The trick is to put a smart pipeline in front of both. As data streams in, this pipeline does the dirty work first: it cleans up the data and tags it with a standard format.

{
  "asset_id": "host-42",
  "user_id": "alice@corp.com",
  "event_type": "process_spawn",
  "source": "CrowdStrike_EDR",
  "timestamp": "2025-07-14T12:05:00Z",
  "raw_payload": { /*...original vendor log...*/ }
}

The result? Your SIEM gets clean, pre-tagged data that’s ready for real-time analysis. It stays fast and focused on finding bad guys but can still pull years of historical logs from the data lake whenever you need to dig deep for a hunt.

You get the best of both worlds: a fast brain (your SIEM) and a perfect memory (your data lake).

Or you can pick one of the next-gen SIEMs that can do all of this for you.

3.Context: The Enrichment Layer

Raw alerts are just noise. So, the platform needs to get all the context.

• External Enrichment: This is your threat intel. Checking IPs, domains, and hashes against VirusTotal or other feeds.

• Internal Enrichment: This is even more important. You need to connect to your own systems, like a CMDB to see who owns the machine, or IAM to see what the user's role is.

The best way is to build these as separate, small services. That way, your SOAR playbooks and your AI agents are calling the exact same enrichment tool. No more duplicate logic.

4.The Correlation Layer

Now that you have alerts and context, you need to connect them. I want to combine multiple detections from the same or different sources in one place. And I want to do that based on those two assets: machines and identities.

For this, you need to mix old-school rules with new-school AI.

• Rules: Handle the easy stuff. "5 failed logins + 1 successful login from a new country = Impossible Travel." Simple.

• AI: After the rules run, let an AI model look at what's left. It can find weird connections that a human or a simple rule would miss.

5.Response Actions: Do Something and Track It

Okay, we have a correlated incident. Now what? I want to be able to do response actions from the same platform. And I need to track the outcomes.

Did that "isolate host" command actually work? Did the user account get disabled? You need a closed loop. The platform has to check and confirm that the action was completed.

6.Emulation: Test Your Work

And if I can also emulate attacks from the same place, it's just perfect. You know the DASEM loop (Detect → Analyze → Simulate → Evaluate → Mitigate). Use a Breach and Attack Simulation (BAS) tool to run tests. See if your detections fire. This is how you find your blind spots before the bad guys do.

The Hybrid Model Is the Only Way

To make all this work, you can't just rely on automation playbooks. And you can't just rely on AI. You need both.

• Rules and SOAR are for the predictable stuff. They are fast and dumb. Perfect for high-volume tasks you see every day.

• AI Agents are for the flexible, complex stuff. They can handle the "what is this?" questions that break traditional automation.

So, How Do You Start?

This sounds big, I know. But you start small.

1. Pick one use case. Phishing triage is always a good one.

2. Build a simple playbook. Ingest the alert, enrich the IOCs, and create a ticket.

3. Add a small AI agent. Have it write a summary for the analyst.

4. Get feedback. Let the analyst rate the AI's summary. This is how it learns.

5. Track your metrics. Mean Time to Respond (MTTR). False positive rate. Analyst hours saved.

6. Iterate. Slowly make it better, then move to the next use case.

Maximizing Your Investment: Beyond API Limitations

Let's talk about a real-world problem. You buy into this "Lego" philosophy, you pick a great platform, but you hit a wall: APIs. Your platform is only as good as the tools it can talk to. What happens when a critical tool in your stack has a terrible API, or worse, no API at all?

This is where the true value of a platform is tested. A good platform doesn't just rely on pre-built connectors. It gives you the power to overcome these limitations. Look for capabilities like:

• Custom Integration Builders: The ability to take any vendor's API documentation—no matter how weird—and build your own integration without waiting for the platform vendor to do it. This puts you in control.

• Going Beyond APIs: Sometimes you need to interact with systems that don't have APIs. Think legacy systems or tools that only have a command-line interface (CLI). A truly flexible platform will let you deploy agents directly on those systems to run commands, parse text output, and feed that data back into your central hub. You're no longer limited by what the vendor supports out-of-the-box.

And this brings up another point: speed. In cybersecurity, things change fast. You might discover you need a new feature or a new type of analysis. The question you should ask any vendor is: "If I request a new feature today, how long until I see it?" A vendor that is truly a partner will have a rapid development cycle, often driven directly by customer requests. If you have to wait six months for a new feature, the attackers have already moved on.

Final Thoughts

At the end of the day, the "single pane of glass" isn't about finding the perfect tool. It's about changing your mindset. Stop looking for a magic box that will solve all your problems. It doesn't exist.

Instead, focus on building a platform that gives you a unified view of your assets and a flexible way to correlate data, add context, and take action. It's a journey, not a destination. You start with one small piece, one Lego block, and you build from there. You measure, you iterate, and you slowly create a system that is tailored to your specific environment and your specific risks.

That is the real single pane of glass. It's not a product you buy. It's a system you build.

This is the only way. You build it piece by piece, like Legos. That's how you get your real "single pane of glass." Not a magic box, but a command center that you built, that you trust, and that actually works for you.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

If you’ve been following my blog, you've probably noticed I preach a lot about the feedback loop. I wanted to do a deep dive on why it's actually important, what we get out of it, and whether it's one of those things everyone talks about but nobody implements right.

My first real go at a feedback loop in cybersecurity was back when I was building the LEAD framework for Threat Intelligence at Adobe. I started because most threat intel programs I saw had no feedback mechanism. The idea was simple: get some Indicators of Compromise (IOCs) and Tactics, Techniques, and Procedures (TTPs), and throw them over the wall to the SecOps team for detection and investigation. End of story. The problem was, I needed metrics to prove the program was working. "No hits based on IOCs" is a terrible metric; I wanted more. I needed to understand if the intel was actually useful or just noise. And yeah, it took a year of hard work, but we got it implemented, and the results were awesome.

Later, when I was running a SecOps engineering team, I figured, why not use the same logic for Detection Engineering and Automation? That’s how I came up with the framework mapping SANS IR stages to Detection Engineering, Standard Operating Procedures (SOPs), and Automation. After that, I created the DSAEM Loop, which breaks it down even further: Detection Engineering > Standard operating Procedures > Automation & Ai Agents > Threat Emulation > Metrics. It makes the whole thing even easier to grasp.

This edition is sponsored by Conifers AI

Achieve SOC excellence with the smart use of AI

Conifers is the AI SOC platform that delivers deep, contextual investigations, adapted to your own data, decisions, and risk tolerance. Continuously learning and adapting to scale your SOC effectiveness and efficiency, Conifers becomes a force multiplier for your team. 

Going to Black Hat? Want to learn more? Let’s meet!

Why Bother With a SecOps Feedback Loop?

Okay, enough storytelling. Why is a well-implemented SecOps feedback loop so critical?

Breaking Down the Process

So, let’s break it down. What SecOps elements do we need to create this loop?

It all starts with the Preparation phase. We build guardrails, put security controls in place, and monitor for threats to keep the bad guys out. First, we build a Threat Profile for our organization, understanding who might target us and what our "crown jewels" are. Once that's sorted, we start collecting logs, starting with our most critical assets and working our way down.

Now for the interesting part: Detection Engineering. This is where we deploy all the detections we need. After that, the alerts go to the SOC team to build runbooks. Then it's back to the automation folks to see what steps can be automated.

Everything is running, alerts are firing, and investigations are kicking off. False positives are going through the roof. Maybe you catch some ransomware before it does real damage, or you get spooked by a false positive IOC related to APT BearPanda12347. You might think everything is working fine. The problem is, if you don't have a good way of feeding this information back into a Lessons Learned process, it's all for nothing. It'll work until it doesn't. And then you're left with broken processes, detections that suck, SOPs no one follows, and automations that never run.

The DSAEM Loop describes this entire lifecycle. It starts with your Detection Engineering, which defines what you're looking for. This feeds into the Standard Operating Procedures that tell your team how to react. Those SOPs are then supercharged by Automation & AI Agents. You validate everything with Threat Emulation, and finally, you wrap it all up with Metrics, which feeds right back into improving your detections. It connects all the dots.

What About the Autonomous (AI) SOC?

Now, many of you are asking how to implement this kind of feedback loop when you're going the full Autonomous SOC route. It’s a great question because throwing AI at a broken process just creates a faster broken process.

Here are the questions you need to be asking any AI SOC vendor:

1. Does the solution understand the detections it’s ingesting? This is key. To understand the context and what to investigate, the AI needs to grasp the detection logic. After the investigation, it needs to provide feedback and recommendations to improve the detection itself. If the AI is just a black box, you’re losing a massive opportunity to tune your defenses.
   

2. How does it handle operating procedures? A one-size-fits-all approach doesn't work. It's important that your SOPs are assigned to your detections and that the platform can access this context to guide its investigation. Some SIEMs let you embed a playbook right in the detection definition, making it easy for an AI SOC platform to use. Can the AI access it, and what will it do with that information?
   

3. What about automation and AI agents? This one is tricky. An AI SOC platform can suggest response actions or even trigger them if it's advanced enough. But this is where you need control. You’ll be the one building the automations or configuring the AI SOC solution to act. You need transparency here. An AI agent that just says "I fixed it" without explaining what it did is a nightmare waiting to happen.
   

4. Can it support threat emulation? This is another critical piece, but let's take it a step further. The real question is, can the AI SOC solution connect to dedicated threat emulation platforms, like the Breach and Attack Simulation (BAS) tools we're all starting to use? This isn't just about manually running a test and checking the logs. An integration allows you to create a powerful, automated validation loop. Your BAS platform executes an attack scenario, and you can measure exactly how your detections, SOPs, and the AI SOC itself performed. Did it catch the multi-stage attack, or did it only see isolated events? Did it correctly correlate the data to identify the threat? This closes the loop between the 'Emulation' and 'Metrics' parts of your DSAEM framework automatically, providing continuous validation of your security posture.
   

How does it handle metrics and the feedback loop? You need a good idea of what you want to measure from the start. Build context into your detections so you can get metrics out. I like to measure the manual effort required for a specific SOP; this helps calculate the added FTE from automation. A good AI SOC should not only provide metrics on its own performance (accuracy, speed, false-positive reduction) but also give you the data to improve your entire security program.

Closing Thoughts

The move to Autonomous SOCs is already happening. But this isn’t about replacing people. It’s about stopping the waste. That Tier 1 role where you just chase alerts all day? That’s going away and it should. It’s not security work, it’s clicking.

We’re finally moving toward real engineering work, building detections, tuning signals, and using automation that actually helps. AI can speed things up, but without feedback and tuning, it just makes the mess faster.

The goal isn’t to replace analysts. It’s to give them space to focus on what matters. Building things. Fixing things. Learning stuff. That’s the SOC we should be aiming for. Not perfect but actually worth showing up for.

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

We've all been chasing the 'single pane of glass' for what feels like forever, right? Another dashboard, another so-called 'solution.' I've been digging into some tech lately MCP, A2A, and AG-UI ,and honestly, it looks like we might finally have the tools to build this thing ourselves. Imagine that.

How I see it is that MCP and A2A will help us fix the integration problem. For a long time, this has been (and still is) a painful process: going through API documentation, building integrations (or requesting one and waiting for a year to get it implemented, or even worse, the vendor tells you they have a voting system, meaning it will never get implemented if none of the big customers want it). And even if you build it, the second challenge is maintaining it; the more API actions you use, the harder it is to maintain.

So, let’s break down what each of these technologies means and how they can contribute to a more unified and intelligent SecOps environment.

Quick Refresher: What Are AI Agents?

An AI agent (or more technically, a group of cooperating, task-specific AI agents) is an intelligent system powered by artificial intelligence, particularly large language models (LLMs), designed to perform specific tasks autonomously or semi-autonomously. Within security operations, these agents significantly enhance the SOC team's efficiency by automating repetitive tasks, augmenting human decision-making, and ensuring consistent, rapid responses to threats. Currently, many of the Agentic AI solutions (also known as Agentic AI SOC Analysts) are focused on automating alert triage and investigation, specifically around Tier 1/Tier 2 investigations. 

As previously discussed in our detailed exploration of SOC AI agents, we outlined four main categories tailored to cybersecurity operations:

1. Tool-Using Agents: Combining LLM reasoning capabilities with external tool integration (e.g., APIs, SIEMs, EDR platforms). They function as "smart SOC assistants" that handle data retrieval, enrichment, and automated actions.

2. Reasoning Agents (ReAct, Chain-of-Thought): These agents explicitly outline their reasoning steps, enhancing transparency and trust in decision-making, critical for compliance-heavy environments.

3. Memory-Enhanced Agents: Equipped with memory capabilities, these agents learn from historical alerts, patterns, and analyst feedback, progressively refining their contextual awareness and reducing redundant analysis.

4. Agentic RAG (Retrieval-Augmented Generation + Autonomy): Advanced agents that autonomously retrieve and synthesise diverse data sources, perfect for complex investigations where multiple context points are essential.

This edition is sponsored by BlinkOps

BlinkOps No-Code Security Agent Builder

The industry's first enterprise platform to build your own custom security agents. Blink enables you to build custom security agents with roles and responsibilities specifically designed to fit your enterprise's unique environment and your security teams needs. Blink Agents deliver a robust Task-Based Automation solution that doesn't sacrifice control. They combine the strengths of AI Agents with the control and security of deterministic workflows.

Decoding the Building Blocks: MCP, A2A, and AG-UI

To understand their collective potential, let's first define each component:

1. MCP (Model Context Protocol): Standardising Agent-Tool Interaction

• Definition: MCP, in the context we're discussing, refers to Model Context Protocol. This is not to be confused with other "MCP" acronyms like Mission Control Platform. Model Context Protocol is designed to standardise how AI models, particularly Large Language Models (LLMs), interact with external tools, APIs, and data sources. Think of it as a universal adapter that allows an AI agent to seamlessly "plug into" various tools it needs to perform tasks. It often uses standards like JSON-RPC 2.0 and enables an LLM to be "fed" context or call a tool by providing a structured way to define tool capabilities and exchange information.

• How it helps SecOps: In cybersecurity, an AI agent equipped with MCP could:

  • Reliably query a threat intelligence platform for the latest on an indicator.

  • Instruct a vulnerability scanner to perform a specific scan and retrieve the results in a usable format.

  • Interact with a SIEM or SOAR platform's API to fetch additional logs or trigger a predefined response action.

  • Gather context from diverse security tools without needing bespoke, brittle integrations for each one. This directly addresses the integration pain points mentioned earlier, as it aims to provide a consistent method for AI to access tool functionalities.

2. A2A (Agent-to-Agent Communication): Enabling Collaborative AI

• Definition: A2A communication establishes a common language and protocol for different AI agents to discover each other, negotiate tasks, share information and context, and collaborate to achieve common goals. This allows for the creation of multi-agent systems where specialized agents can work together, even if they are developed by different vendors or for different primary purposes. These protocols often ensure secure and structured data exchange, supporting features like context passing, stateful interactions, and permission controls.

• How it helps SecOps: A2A is crucial for building a truly intelligent and autonomous SOC. Imagine:

  • A "Triage Agent" that performs initial alert enrichment could use A2A to pass its findings to a specialised "Reactive Threat Hunting Agent".

  • The Threat Hunting Agent, after uncovering deeper threats by querying various tools (potentially via MCP), could use A2A to coordinate with a "Response AI Agent" to execute containment actions like isolating a host or blocking an IP address.

  • Different agents specialising in IOC enrichment, asset context, and malware analysis could share their findings in real-time to build a comprehensive understanding of an incident much faster than a human analyst working alone or a monolithic automation script.

3. AG-UI (Agentic User Interface): Crafting the Human-AI Partnership

• Definition: An Agentic User Interface (AG-UI) facilitates natural and effective interaction between human users and AI agents (or systems of AI agents). Instead of just a dashboard displaying data, an AG-UI allows users to converse with agents, delegate tasks using natural language, receive proactive suggestions, and understand the reasoning behind agent decisions and actions. It's about creating a collaborative workspace where humans and AI can augment each other's capabilities. It focuses on explainability, interactivity, and allowing users to supervise and guide AI agents effectively.

• How it helps SecOps: This is where the "single pane of glass" vision truly comes to life:

  • SOC analysts could interact with a suite of security agents through a unified, conversational interface. Instead of manually querying multiple tools, they could ask: "What's the latest on the phishing attempt from this morning? Has the source been blocked, and have we seen similar attempts?"

  • The AG-UI could present a consolidated view of an incident, not just raw data, but a narrative pieced together by collaborating agents, showing the steps taken, the rationale, and proposed next actions. Analysts could then approve, modify, or query these actions.

  • It can help demystify complex AI processes by providing transparency into agent operations , building trust and enabling analysts to effectively supervise AI-driven tasks.

The Convergence: Towards an Intelligent, Unified SOC

The real magic happens when MCP, A2A, and AG-UI work in concert:

• MCP handles the "how" of tool use: It allows individual agents to reliably interact with the diverse set of security tools in the SOC's arsenal – from EDRs and firewalls to threat intel feeds and vulnerability scanners. This solves the fundamental integration challenge.

• A2A manages the "who" and "what" of collaboration: It enables these specialized agents, each proficient with certain tools via MCP, to communicate their findings, share context, and coordinate complex workflows. For instance, an "IOC Enrichment Agent" uses MCP to query VirusTotal, then uses A2A to share the results with a "Timeline Analysis Agent" and an "Incident Coordinator Agent".

• AG-UI provides the "where" and "why" for human oversight: It offers a centralized interface where human analysts can understand the bigger picture assembled by these collaborating agents, drill down into specifics, provide judgment, and steer the overall response. This makes the advanced automation accessible and trustworthy.

This combination directly tackles the desire for a "single pane of glass" not by just aggregating dashboards, but by creating an interactive, intelligent system that automates much of the underlying data gathering, correlation, and even decision-making, while keeping humans firmly in the loop for critical judgment and strategic oversight.

Analogy to Human SOC Teams:

Think about how a human SOC team operates:

• Analysts have their specialized tools (SIEM, EDR console, TI portal) – MCP ensures AI agents can also use their "tools."

• Analysts communicate with each other during an investigation (e.g., a Tier 1 analyst escalates to Tier 2, who consults a malware specialist) – A2A enables AI agents to have similar collaborative dialogues.

• The SOC manager or lead analyst has an overall view, receives summaries, and makes critical decisions – AG-UI aims to provide this interactive, supervisory layer for human operators overseeing AI agents.

Practical Implications for Security Automation

Integrating AI agents using this trifecta can move SecOps beyond rigid, rule-based playbooks towards more adaptive, context-aware automation.

• Reduced Integration Burden: MCP’s standardized approach to tool interaction could significantly cut down the time and effort spent building and maintaining custom integrations for every new tool or API change.

• Smarter Enrichment: Instead of a 10-step enrichment playbook that often breaks , an agent (or a team of agents using MCP and A2A) can dynamically figure out what data it needs, fetch it from various sources, and synthesize a much richer contextual picture.

• Dynamic Triage and Prioritization: A "Triage Classifier Agent" could use learned patterns and real-time context (gathered via MCP and shared via A2A with other intelligence agents) to more accurately prioritize alerts, reducing alert fatigue.

• Automated Threat Hunting Loops: A "Reactive Threat Hunting Agent" could be triggered by a high-confidence alert, use MCP to query historical data across multiple platforms, identify patterns, and even suggest new detection rules or areas for proactive hunting, all while informing the human analyst via the AG-UI.

• Streamlined Incident Response: From initial detection and enrichment through investigation, containment, and eradication, different AI agents can take on specialized tasks, coordinating via A2A and presenting a unified view and control points to human responders through the AG-UI. This is evident in the blueprint discussed for AI agents handling alert reception, reactive threat hunting, and response actions.

The Challenges Ahead

While the vision is compelling, it's not without hurdles:

• Standardisation: The success of MCP and A2A relies on widespread adoption of common standards. While initiatives exist, achieving industry-wide consensus takes time.

• Security of the Agents Themselves: If agents can take action, they become targets. Securing the agents, their communication (A2A), and their tool interactions (MCP) is paramount. This includes addressing risks like data leakage, unauthorised tool access, and potential manipulation of agent decision-making.

• Complexity: Building, managing, and debugging systems of interconnected, autonomous agents can be complex.

• Trust and Transparency: Analysts need to trust these systems. AG-UIs must be designed to provide transparency into agent reasoning and actions ("opaque decisions" are a major pitfall ), and robust guardrails and human oversight mechanisms are crucial.

• Training and Cultural Shift: SOC teams need to be trained not just on new tools, but on how to collaborate effectively with AI teammates. This involves teaching prompting, delegation, and supervision skills.

Final Thoughts: The “Build or Buy” Moment for SecOps

The potential here is massive. For the first time, it feels like we have the right building blocks MCP, A2A, and AG-UI to finally crack the integration and automation problem without just adding another layer of complexity. The conversation is shifting from "if" we can automate to "how" we do it intelligently.

This brings every SecOps team to a crossroads: do you start piecing these technologies together yourself, or do you look for a platform that has already harnessed this power?

Building it in-house gives you ultimate control, but it's a heavy lift. On the other hand, a new breed of security automation platforms is emerging that's built on these very agentic principles. They offer a way to get the power of AI agents without the massive R&D effort, giving teams a launchpad into AI-first SecOps.

The key takeaway is that you're no longer stuck with the old, rigid playbooks of the past. Whether you build or buy, the future of the SOC is intelligent, adaptive, and agent-driven.

Have opinion on this, join the discussion here!

Vendor Spotlight: BlinkOps

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

If you've been around this blog for a while, you know we love cutting through the noise. Last time we geeked out over the shift from rule-based playbooks to adaptive AI agents. Today we’re diving into something even messier: how do we measure ROI and KPI when it comes to AI tools within Security Operations?

Spoiler alert: It’s not just about how many alerts they auto-close.

Yeah, I get it, everyone wants clean dashboards, KPIs, some might still want that pew pew cyber map (yes, they are still a thing: https://threatmap.checkpoint.com/).

But here’s the uncomfortable truth: closing alerts doesn’t mean your SOC is getting smarter. It just means you’re sweeping faster. And with a whole AI and Agentic staff, it's no longer just about how fast you process alerts. Yes, you still want to show that you now close alerts in 15 minutes rather than 1 hour, but in the end, I don't care if the Autonomous (AI) SOC solution closes the alert in 10 seconds or 60 seconds. Now we are in an era of trust; I'm interested in how accurate the analysis was, why it involved a human, why it closed a false positive, and whether it fed back to detection engineering for improvement.

What really matters is whether your AI agents are helping you improve over time (please don't measure how they replace you over time, that won't work and we are not there yet; remember when SOAR promised it would automate and replace analysts? Ten years later, we still didn't win that metric).

So, because I like to invent terms (because in cybersecurity we will never stop doing that, and I realized I'm not the one who will change it), I’ve mapped out how to measure if your AI tools are actually pulling their weight across the entire security incident lifecycle , from preparation all the way through to learning and adapting. I’m calling this framework the PICERL Index for Autonomous (AI) SOC. PICERL (that’s P-I-C-E-R-L) stands for Preparation, Identification, Containment, Eradication, Recovery, and the critical Learning phase. The whole idea behind this Index is to break down when to use which metric, so you can see if your AI SOC is truly getting smarter, not just faster.

This edition is sponsored by Prophet Security

How’s AI Really Being Used in the SOC?
Tell Us

We’re running a quick 8-10 minute survey to find out how teams are actually using AI in the SOC. The first 100 qualified respondents get a $50 Amazon gift card, and everyone gets early access to the full report.

Help shape one of the most forward-looking industry benchmarks on AI in security operations.

Why take it?

- See how your peers are using AI in their SOC
- Receive the full anonymized report before public release
- $50 Amazon gift card for the first 100 qualified respondents

Complete the survey now

I’ve created a dedicated page where you can get an overview of all the metrics.
Check it out: https://reports.cybersec-automation.com/picerl-index-ai-soc

Newsletter Recommendations

Sponsored

SheHacksPurple Newsletter

Learn to Code Securely, with Tanya Janca

Subscribe

Disclaimer: Opinions expressed are solely my own and do not reflect the views or opinions of my employer or any other affiliated entities. Any sponsored content featured on this blog is independent and does not imply endorsement by, nor relationship with, my employer or affiliated organisations.

If you've been following the blog, this post is a natural continuation of where we left off in AI Agents vs. Automation Playbooks. In this piece we dug into the philosophical split between rigid, rule-based playbooks and adaptive, context-aware agents. TL;DR: playbooks are great at executing instructions, but AI agents can think. 

One of the questions I keep getting, and honestly something I’m neck-deep working on right now, is how do we actually transition from classical playbook automation to an autonomous SOC? And here’s the deal: the biggest challenge isn’t tech. It’s finding the right balance. And when I say balance I don’t mean those high-level "strategic framework" slides with five circles and a fancy acronym (guilty as charged, that’s most of what I post). What I mean is: take those frameworks and actually turn them into real, tactical stuff. Use-case breakdowns, step-by-step flows, actual hands-on implementations. That’s how you move from theory to value.

Like let’s be honest, you’re not going to just swap out your entire phishing playbook with an AI agent and call it a day. It’s not plug-and-play magic. You need to break it down step by step. Map out each stage. Ask: what needs control? What needs judgment?

That’s also where trust comes in. I’ve heard way too many stories where teams replaced a playbook with an AI agent, ran it once, got mixed results (because duh, nothing just works first try in tech), and then immediately wrote it off as immature tech. The vibe is always, "This sucks, let's wait a few more years."

WRONG.

What they missed was the process. If you treat it like a drop-in replacement, you’re setting it up to fail. But if you audit the structure of your playbooks, and slowly infuse agents where it makes sense, you'll get two big wins: 1) gradual, trackable progress, and 2) analyst buy-in, because now they can see where it helps and where human eyes are still needed.

Done right, you buy analysts back the one thing you can’t scale: time. Done poorly, you add noise, risk, and spark a full-on mutiny in the SOC.

This edition is sponsored by Prophet Security

Empower your SOC with a Force Multiplier

Security alerts overwhelm SOC teams with low-value noise, draining analyst time and slowing response. The result is alert backlogs, analyst fatigue, and missed opportunities to improve security outcomes.
Prophet Security’s agentic AI SOC Analyst removes that burden by autonomously triaging and investigating every alert in under three minutes. Analysts stop wasting time on false positives or context switching between disconnected tools. Instead, they focus on higher-value work like threat hunting, incident response, and building better detections.

See how Prophet AI works >