As AI takes on more security operations center (SOC) workflows to automate threat triage, indicator extraction, and incident report generation, security operations leaders face a persistent question: Does SOC performance depend more on the large language model (LLM) deployed or on the underlying quality of the security data it consumes?
Academic studies offer conflicting answers. An original research article from the Frontiers in Artificial Intelligence emphasizes output variation in accuracy, relevance, and clarity across frontier models such as Claude 3.5 Sonnet, Gemini, ChatGPT 4o, and Mistral Large 2. Meanwhile, published work in the Information Systems journal argues that “trustworthy AI applications require high-quality training and test data along many quality dimensions, such as accuracy, completeness, and consistency.”
Recent research favors the data argument by demonstrating that data quality rather than model selection can be a significant factor in governing security operations success. According to the findings, high-fidelity network evidence can improve security outcomes by 2-4x across core investigation metrics.
For CISOs, these outcomes offer direct operational advantages:
- Provide concrete telemetry, which reduces mean time to respond (MTTR)
- Control token expenditure, which helps control cost
- Ensure higher return on security investments, which helps prove security team efficacy
Security practitioners can evaluate the findings to guide future SOC architecture decisions, and CISOs can use these findings to justify infrastructure investments, lower analyst turnover caused by alert fatigue, and provide defensible, demonstrable security metrics to executive teams and board members.
Putting the question to the test
To measure the true drivers of AI performance in enterprise security environments, the Provably Better Data research project established a controlled test framework. The project evaluated model performance across two operational benchmarks:
- Capture the flag (CTF) scenario: a 44-question investigation based on a Volt Typhoon attack campaign
- Incident response analysis: a report generation task based on a Salt Typhoon dataset
To isolate data quality as the single variable, the test harness processed four distinct network telemetry sources under identical conditions:
- Corelight enriched logs
- Open source nDPI firewall logs
- Snort 3 intrusion detection system alerts
- NetFlow connection telemetry
To ensure the test was accurate and repeatable, each dataset was run multiple times via an Open Cybersecurity Schema Framework (OCSF) normalized schema. The LLMs used were Anthropic Claude Opus 4.6, Google Gemini Pro 3.1 Preview, and older models from both providers. Each model was independently tested with identical prompts across all test runs, and the models were scored on CTF accuracy and incident response (IR) claims supported by available evidence.
Findings: The evidence ceiling in automated workflows
Frontier language models provide advanced reasoning capabilities, but the quality of their conclusions remains bounded by the available evidence. When telemetry lacks detailed protocol-level context, AI agents cannot infer what was never collected. The investigation can only go as far as the data allows.
Consider this CTF question and answer from the Corelight data compared to the nDPI response. Agents were asked to address a query regarding the NetBIOS computer name for IP address 10.110.154.113. Their performance depended entirely on log context:
- Corelight logs provided the correct answer, FINANCE01, which is found in the server_nb_computer_name field in Corelight’s NTLM log. Zeek parsed the NTLM Type 2 challenge message and extracted the server’s computer name.
- Firewall logs identified NTLM protocol activity but failed to parse individual fields within the NTLM challenge, leaving it unable to return the correct answer.
These telemetry gaps can create significant operational friction. Although firewall logs provided general connection visibility, their missing protocol fields left the investigation incomplete. When evidence is missing, human analysts must manually verify automated outputs, but when it’s there, AI can deliver answers they can act on immediately.
Better data can yield 2-4x better outcomes
Observed measurements from the project illustrated that superior data fidelity yields 2-4x superior security outcomes compared to standard logs.
In the 44-question Capture the Flag (CTF) benchmark, accuracy rates varied dramatically based on source data:
- Corelight logs: 95.2% accuracy rate
- Firewall logs: 58.3% accuracy rate
- Snort 3 alerts: 39.4% accuracy rate
- NetFlow records: 25.8% accuracy rate
Access to richer telemetry had a measurable impact on model performance. Corelight enabled each LLM to answer all 44 questions with direct log evidence, whereas NetFlow supported only 15 questions. As a result, Corelight achieved a CTF score of 4,178.3 points, compared with 970.0 points for NetFlow, a more than fourfold improvement.
The incident response experiment revealed similar gaps across overall coverage, total rubric scores, and critical findings:
- Corelight logs: 90.3% evidence coverage rate
- Firewall logs: 61.3% evidence coverage rate
- NetFlow records: 30.9% evidence coverage rate
- Snort 3 alerts: 21.2% evidence coverage rate
For Tier 1 critical investigation requirements, Corelight logs enabled models to answer 91.7% of the mandatory questions, whereas NetFlow logs supported only 18.3% of these requirements and Snort 3 alerts supported only 10%.
The outcome: High-quality data yielded a fivefold increase in critical incident visibility over basic flow records.
Investigation speed also improved significantly with the enriched data. The LLM completed the full investigation in 14.7 minutes when provided with Corelight logs. The same model required 27.0 minutes with NetFlow logs and 26.3 minutes with firewall logs.
The outcome: Lower-quality data nearly doubled investigation times because models entered repeated retry loops.
It’s also important to note that the LLMs rarely generated hallucinated outputs when prompts instructed them to mark missing evidence as unanswerable. Hallucination counts remained at zero (0) for Corelight, firewall, and NetFlow datasets, while Snort 3 alerts recorded 1.9 hallucinations. Grounded models prevent analysts from chasing fabricated evidence, which reduces investigation time and improves confidence in the results.
Recommendations for security leaders
AI automation can accelerate threat detection, indicator tracking, and incident containment across enterprise networks. Effective automation, however, doesn’t happen automatically. It requires complete, structured, and protocol-aware telemetry to achieve reliable outcomes.
According to the research, model upgrades and complex prompt engineering will not overcome fundamental data deficiencies. Given the outcomes of this experiment, SOC leaders who plan future operations center investments should strongly consider prioritizing evidence quality over model selection.
For more in-depth analysis, review the detailed methodology, agent architecture, and complete experimental metrics in the Provably Better data white paper on Corelight’s website.
Corelight Network Detection and Response
Better data can boost security outcomes by 2-4x. Learn why high-fidelity network evidence is the prerequisite for AI-driven security. Corelight: Defending the world’s most sensitive networks. Learn more.
