Unpredictable Behaviour of AI Agents: An Emerging Cybersecurity Concern

Recent disclosures involving OpenAI, Anthropic, Meta and the UK’s AI Security Institute (AISI) have raised concerns about the unpredictable behaviour of increasingly autonomous AI agents during cybersecurity testing.

What are AI Agents?

• AI agents are autonomous AI systems designed to pursue specific goals and perform tasks with limited human intervention.
• Unlike conventional chatbots that mainly respond to prompts, AI agents can make decisions, determine a sequence of actions and interact with external systems.
• They may perform tasks such as reading and sorting emails, analysing financial data, accessing websites or using software tools.
• Their greater autonomy makes their behaviour more difficult to predict and increases the importance of testing before deployment.

Why are AI Agents Different from Traditional Software?

• Traditional software generally performs predefined actions, whereas AI agents may independently decide how to achieve a given objective.
• They are increasingly being authorised to act on behalf of users.
• Errors, manipulation or unexpected behaviour can therefore produce real-world consequences, rather than remaining confined to a chatbot conversation.

Major Security Risks Associated with AI Agents

• Prompt Injection: Hidden malicious instructions in webpages, documents or other content can manipulate an AI agent’s behaviour.
• Reasoning and Planning Failures: Errors in decision-making may cause an agent to pursue unintended objectives.
• Excessive Tool Permissions: Access to external tools or compromised software may result in unintended actions such as sending emails or modifying code.
• Interconnected-System Risk: Agents interacting with websites, software services or other AI agents may spread risks across multiple connected systems.

Cybersecurity vs AI Alignment Problem

• Traditionally, cybersecurity focuses on protecting computer systems from human attackers using technology as a tool.
• AI agents create a new situation where the system itself may take actions that developers neither explicitly instructed nor anticipated.
• Some researchers classify such incidents as alignment failures rather than conventional cybersecurity failures.
• Capability failure occurs when an AI cannot complete the assigned task, whereas alignment failure occurs when it pursues an objective in ways that violate intended constraints.

Key Concerns

• Increasing autonomy can make agent behaviour difficult to predict.
• Absence of a human-in-the-loop can allow unintended actions to cause real-world harm.
• Greater access to software, networks and infrastructure increases the potential scale of damage.
• Testing conducted only after deployment may be inadequate for highly autonomous systems.

Way Forward

• Conduct rigorous pre-deployment cybersecurity and alignment evaluations.
• Use independent evaluators to assess AI systems at earlier stages of development.
• Apply the principle of least privilege, limiting agents only to tools and permissions necessary for their task.
• Strengthen safeguards against prompt injection and manipulation.
• Introduce continuous monitoring and human oversight for high-risk actions.
• Develop stronger standards and regulatory frameworks for autonomous AI systems.

As AI agents gain greater autonomy and access to real-world digital infrastructure, the boundary between AI safety and cybersecurity is increasingly converging. Their safe deployment will require not only protection from external attackers but also mechanisms to ensure that the agents themselves remain predictable, controllable and aligned with human intentions.