Why Most AI SOC Tools Fail, and What Actually Delivers ROI: Monzy Merza at Black Hat USA 2026
Interview by Shira Rubinoff, CEO of the Cybersphere Group, live at Black Hat USA 2026
AI is routinely pitched as the fix for analyst burnout and triage fatigue, but the day-to-day reality inside the SOC is messier than the marketing. Live at Black Hat USA 2026, Shira Rubinoff sat down with Monzy Merza, Crogl's Founder and CEO, to ask where AI is actually delivering measurable ROI for SOC teams today, and where current tools are failing, causing friction, or introducing new operational risk.
Where AI SOC Tools Actually Fail
Merza breaks the customers and prospects Crogl talks to into three groups. The first is skeptical: they tried building something on AI themselves and it did not work. The second bought a product that did not work operationally, usually because it was SaaS-driven or disconnected from their actual environment. The third is starting to see real value, either because they invested heavily in an in-house build or because they found a tool that combines the sovereignty and privacy their environment requires with the operational relevance of the work analysts do every day.
That third group is the exception, not the rule. Most failures trace back to the same two problems: the tool cannot be governed the way the organization needs it governed, or it does not actually plug into how analysts work day to day.
The Analyst Becomes an Engineer, Not a Dashboard-Watcher
As AI moves from an experimental assistant to an embedded layer of everyday operations, Merza argues the tooling and the workflows underneath it both have to change, not just the surface. One customer put it directly: if an AI-assisted analyst is expected to do more, that analyst needs to be treated more like an engineer and less like someone who stares at dashboards all day.
That shift is showing up as a concrete product request. Customers are asking Crogl to make its interface look less like a rigid dashboard with predetermined, click-through workflows and more like an integrated decisioning environment, a kind of IDE where the analyst works the way they actually want to work. Merza's framing for why incremental change does not get you there: "Nobody can build a rocket by iterating over a horse. You have to take a transformative leap to do something significantly different."
Sovereignty, Workforce, and the New AI Attack Surface
Asked whether AI is a genuine force multiplier or something that fundamentally restructures how security teams are staffed and budgeted, Merza pointed to three concerns that come up consistently with large, complex customers: utilities, banks, and government agencies among them.
First is sovereignty: whether the organization can govern the AI and keep it private, on their terms, regardless of how capable it is. Second is the workforce question: whether the tool helps people do what they already know how to do, rather than assuming AI will substitute for domain expertise nobody actually lacked. Third is operational relevance: the on-prem and cloud footprint security teams already protect has not gone away, and now there is a new AI-driven footprint layered on top of it that also needs protecting. As Merza put it, the story that gets missed is that this isn't just about using AI. It's also about protecting the AI that's now part of the business.
Is AI Actually Lowering Costs?
On whether AI is genuinely reducing the cost of vulnerability management or simply shifting where the money goes, Merza frames it as an implementation question, not a question about whether the underlying pattern works. A lot of vendors, in security and elsewhere, are effectively reselling tokens: a thin wrapper on top of a frontier model, sold as a product. In those cases, results are negligible at best and negative at worst, in line with the MIT report from late last year finding that roughly 95% of AI projects fail, largely because they are not anchored to concrete outcomes.
Where the pattern does work is when an organization has a clear set of use cases and realistic expectations going in. Merza pointed to a customer's "cyber task order" process (reviewing 20-to-30-page threat intel advisories) that used to take about two weeks to work through. Using Crogl to interpret the document and execute the threat hunt compresses that to under an hour. Not the "seconds" pitched on a demo floor, but a real multi-week task collapsed into a sub-hour time horizon.
Why Crogl Made Its AI SOC Platform Free
Asked about the decision to launch a free, enterprise-grade AI SOC platform, Merza traced it back to Crogl's founding team's background building software at Microsoft, Splunk, and Databricks. Rather than compete in an increasingly crowded AI SOC market on feature-list battles, the team decided to let the community decide for itself. Teams that need to solve a real problem shouldn't have to sit through a long sales cycle or a dozen calls before they can get their hands on the product.
The free tier is meant to remove that friction entirely: download it, try it, and join Crogl's community Slack to talk directly with the engineers building it.
Frequently asked questions
Why do most AI SOC tools fail?
Most failures fall into two categories: tools built or bought that never worked operationally because they were SaaS-driven or disconnected from the customer's actual environment, and tools that could not be governed the way a sovereignty-sensitive organization required. The exceptions are tools that combine strict data sovereignty and privacy with genuine day-to-day operational relevance.
How is the security analyst's role changing as AI adoption grows?
Customers are asking AI SOC vendors to treat analysts more like engineers than dashboard-watchers, which means the tooling itself has to change, not just get a new coat of paint. The product shift underway is toward an integrated decisioning environment, closer to an IDE than a fixed dashboard with predetermined, click-through workflows, where analysts can work the way they actually want to work.
Is AI actually lowering the cost of security operations?
It depends on implementation, not on whether the underlying approach works. Vendors reselling a thin wrapper around a frontier model tend to produce negligible or negative results, consistent with an MIT finding that around 95% of AI projects fail due to a lack of anchored outcomes. Where organizations define clear use cases and realistic expectations, the gains are real: one Crogl customer cut a threat-intel review process from about two weeks down to under an hour.
Why did Crogl launch a free enterprise-grade AI SOC platform?
Rather than compete purely on feature lists in a crowded market, Crogl's founding team, with backgrounds at Microsoft, Splunk, and Databricks, decided to let the security community try the platform directly instead of gating it behind a long sales process. The free tier removes that friction: download it, use it, and talk to Crogl's engineers directly in the community Slack.