Loading story…
Large Language Models (LLMs) are increasingly being integrated into Security Operations Centers (SOCs) to enhance productivity and automate routine tasks such as transcribing meetings, extracting action items, and prioritizing emails. However, significant challenges remain before LLMs can reliably automate the majority of SOC functions, particularly those requiring high precision and consistent execution across vast, real-time data streams. Current LLMs struggle with real-time ingestion at scale, as they must process rapidly accumulating data from diverse sources like logs, EDR, cloud resources, and code without lag or data loss. Another major limitation is the need for large, durable context retention, enabling the system to interpret actions and sequences over extended periods using historical knowledge such as asset inventories and case histories. Low-latency, low-cost execution is also essential, as LLMs must filter, correlate, enrich, and reason over incoming data at enterprise scale without incurring prohibitive costs. Deterministic logic is required for repeatable, reliable results, but LLMs often lack this capability, making them less suitable for critical SOC triage tasks that demand context assembly, hypothesis generation, and nuanced business-risk judgment. Meanwhile, as attackers increasingly weaponize AI-driven automation, defenders must achieve a balance where machines can thwart most attacks at machine speed, or risk falling behind. In addition to operational challenges, LLMs introduce new security risks, such as system prompt leakage, which is now recognized as a top concern by the OWASP Top Ten Risks for LLMs. System prompt leakage occurs when sensitive information embedded in the prompts used to instruct AI models is inadvertently exposed, potentially revealing credentials, permissions, or internal logic to attackers. This exposure can enable adversaries to understand system guardrails, internal rules, or filtering criteria, making it easier to bypass defenses or launch targeted attacks. To mitigate this risk, experts recommend never embedding sensitive data such as passwords or internal logic within system prompts, ensuring that even if a prompt is leaked, it does not provide attackers with actionable insider knowledge. Examples of prompt leakage include revealing database storage details, internal decision-making processes, or request filtering limitations, all of which can be exploited by attackers. The convergence of these operational and security challenges underscores the need for robust, secure LLM deployment strategies in SOC environments. Organizations must address both the technical limitations of LLMs and the emerging risks associated with their use, particularly as threat actors become more sophisticated in exploiting AI-driven systems. Effective SOC automation will require not only technological advancements in LLM reliability and scalability but also rigorous security practices to prevent information leakage and maintain operational integrity. As the adoption of LLMs in cybersecurity accelerates, continuous evaluation of both their capabilities and vulnerabilities will be critical to maintaining a resilient defense posture.
Mallory correlates global threat intelligence with your attack surface — know if you’re exposed before adversaries strike.
3 events from the most recent confirmed update back to the earliest known activity.
Cisco Talos researchers reported that reusing the same LLM session for multiple breach reports can cause details from one incident to bleed into another, even after prior notes are deleted. Their testing across ChatGPT, Claude, and Gemini found inconsistent outputs and session memory effects, leading Talos to recommend human validation for AI-assisted incident reporting.
Help Net Security published an article arguing that GPT-based systems used in security operations automation need to be reworked for security, reflecting broader concern about LLM security and operational use.
FireTail published a blog post focused on 'LLM07: System Prompt Leakage,' highlighting system prompt leakage as a distinct security issue in large language model deployments.
3 references tracked. Mallory keeps watching after this page renders.
govinfosecurity.com
Open sourcehelpnetsecurity.com
Open sourcesecurityboulevard.com
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Organizations deploying large language model (LLM) applications face significant security and operational risks, including unbounded resource consumption, novel attack vectors, and the need for advanced anomaly detection. Attackers can exploit LLMs by submitting massive, compute-intensive requests, leading to "denial of wallet" attacks that can drain cloud budgets and disrupt business operations. The OWASP Top 10 for LLMs highlights unbounded consumption as a critical vulnerability, emphasizing the importance of implementing resource controls and monitoring usage patterns to prevent financial and service impacts. Additionally, the Model Context Protocol (MCP) introduces new security challenges, as traditional rule-based and signature-based systems are inadequate for detecting sophisticated, context-dependent threats targeting LLM infrastructure. To address these evolving risks, security teams are adopting AI-driven anomaly detection and exposure management strategies that prioritize real, exploitable risks over alert volume. The shift from reactive monitoring to proactive observability and context-aware security is essential for protecting LLM-powered platforms. As threat actors increasingly leverage LLMs to enhance their campaigns, defenders must invest in specialized, security-focused LLMs and scalable infrastructure to keep pace with adversaries and safeguard critical AI assets.
Jun 29, 2026
The widespread adoption of large language models (LLMs) in enterprise environments has introduced significant security challenges, particularly due to the tendency to over-rely on their outputs and the normalization of risky behaviors. Experts warn that treating LLMs as reliable and deterministic can lead to systemic vulnerabilities, as these models are inherently probabilistic and can be manipulated through techniques such as indirect prompt injection. This normalization of deviance—where unsafe practices become accepted due to a lack of immediate negative consequences—mirrors historical safety failures in other industries and is exacerbated when vendors make insecure design decisions by default. In addition to technical risks, human factors play a critical role in LLM security. Employees may inadvertently expose sensitive data by pasting it into public LLMs, blindly trust AI-generated outputs, or bypass security policies for convenience, making internal misuse a primary concern. While technical controls such as AI governance and access restrictions are important, organizations must also prioritize security awareness training to address the human side of LLM risk. Building a culture of responsible AI use is essential to mitigate both external threats and internal errors associated with LLM deployment.
Jun 29, 2026
The widespread integration of large language models (LLMs) into enterprise environments is introducing new security risks at every layer of the technology stack. Security leaders are being urged to rethink traditional trust boundaries, as LLMs can alter assumptions about data handling, application behavior, and internal controls. Key risks include prompt injection, sensitive data leakage through inputs and outputs, and fragmented ownership of LLM-related security responsibilities. Experts emphasize the need to treat LLMs as untrusted compute and to enforce explicit policy and validation layers, rather than relying solely on prompt engineering or fine-tuning. Meanwhile, cybercriminals are actively exploiting the popularity of LLMs by selling discounted access to mainstream AI tools such as ChatGPT, Perplexity, and Gemini on underground forums. These tools are being used by threat actors for a range of malicious activities, including phishing, reconnaissance, and automating cybercrime operations. The criminal use of LLMs lowers the barrier to entry for less-skilled attackers and enables more efficient execution of threat campaigns, highlighting the dual challenge of securing enterprise LLM deployments while monitoring their abuse in the cybercriminal ecosystem.
Jun 29, 2026