How AI and Bots Are Transforming Web Security: Insights from Arcjet CEO David Mytton

The web is experiencing a fundamental transformation as bots and AI agents become a significant part of online traffic. According to Arcjet CEO David Mytton, around half of all internet traffic now comes from automated sources, and this proportion is only set to increase as new agent technologies become faster and more widely adopted. This shift brings both opportunities and challenges for developers and organizations aiming to secure their applications.

The Changing Nature of Bot Traffic

In the past, blocking bots was often handled with simple rules, such as filtering by IP address or user agent. These methods were imprecise and often resulted in legitimate traffic being blocked alongside malicious requests. Today, the landscape is more complex. Some bots and agents act on behalf of real users, performing valuable tasks like searching for information, making purchases, or indexing content for search engines. Others may attempt to scrape data or abuse APIs.

Blocking all bots is no longer a practical solution. For example, some AI crawlers can increase site signups and drive genuine business value. Instead, developers need to understand the intent and context of each automated request, allowing beneficial bots while restricting harmful ones.

The Importance of Application Context

Modern security requires a nuanced approach. Application context is critical for making informed decisions about which traffic to allow or deny. For instance, an e-commerce site risks losing revenue if it blocks a legitimate transaction simply because it appears automated. Instead, suspicious activity might be flagged for human review rather than blocked outright.

The context of the request; such as the user’s session, the part of the application being accessed, and the behavior of the client; should all inform security decisions. This level of detail helps distinguish between helpful automation and abuse.

Use case: The High-Stakes Challenge of Agentic Payments

Enabling AI agents to make payments on behalf of users introduces a new frontier of challenges, as the global financial system was designed for humans, not machines. This fundamental mismatch creates significant hurdles:

Infrastructure and Security: Today's payment infrastructure is filled with anti-bot mechanisms like CAPTCHA and two-factor authentication that deliberately block automated use. This creates a vastly expanded attack surface, with new threats like prompt injection to manipulate an agent's intent or model-in-the-middle attacks to steal data.

Identity and Fraud: There is no reliable system to verify an agent’s identity, a problem termed "Know Your Agent" (KYA). Existing fraud detection tools are tuned for human behavior and may incorrectly block legitimate agent purchases, while fraudsters can design sites specifically to trick agents.

Regulatory and Liability Gaps: Critical legal questions remain unanswered. If an agent makes a mistake or overspends, it's unclear whether the user, the AI provider, or the merchant is liable. Regulations like the EU's Strong Customer Authentication (SCA) and PCI compliance standards were not designed for autonomous transactions, creating major legal ambiguity.

Trust and Integration: Users must place their financial trust in "black box" AI systems, creating a significant adoption barrier. Technically, integrating with thousands of different payment providers and complex checkout flows is a monumental challenge for developers.

Evolving Standards and Controls

Tools like robots.txt have long provided a voluntary way to signal to bots which parts of a site they can access. Good bots, such as those from search engines, typically respect these signals, while others may ignore them or even use them to find sensitive areas of a site. This creates a control problem for site owners, who need to enforce rules and manage access more effectively.

Fingerprinting techniques are becoming more sophisticated, allowing developers to identify and manage automated clients based on a combination of signals like IP reputation, user agent strings, and session characteristics. Reverse DNS lookups and public key cryptography are also being explored to verify the identity of bots and agents.

Strategies Builders Are Adopting

Developers and security teams are implementing a range of strategies to address these challenges:

Layered Defense: Combining Network and Application-Level Protections

Layered defense represents the gold standard for bot and agent security, combining multiple security mechanisms to create comprehensive protection. This approach recognizes that no single security product can fully safeguard a network from every attack it might face.

Modern implementations include several key components:

• Network-level protections using Web Application Firewalls (WAFs), CDNs, and firewalls for initial filtering

• Application-level context that analyzes request patterns, user sessions, and business logic

• Behavioral analysis that monitors interaction patterns and identifies anomalies

Leading security providers like F5, Radware, and Fortinet implement sophisticated multi-layered approaches that use AI-driven detection, behavioral analysis, and real-time threat intelligence. These systems employ progressive challenge mechanisms including JavaScript challenges and CAPTCHA to verify human users while maintaining performance.

The effectiveness of layered defense lies in its redundancy - if one security layer is compromised, other measures can limit and mitigate damage to the entire network.

Granular Access Controls for Bots and Agents

Granular access control (GAC) enables organizations to specify exact actions users can perform on given resources based on their roles and responsibilities. This approach follows the principle of least privilege, providing users with only the minimum access necessary for their tasks.

Key implementation strategies include:

• Role-Based Access Control (RBAC) as the most granular type of access control

• Attribute-Based Access Control (ABAC) for context-aware authorization

• Fine-grained filtering that prevents unauthorized AI data access through granular policies

• Real-time enforcement that dynamically authorizes requests before they reach AI models

  
  

For AI agents specifically, modern frameworks like Permit.io provide fine-grained permissions across AI prompts, responses, actions, and data access. This includes prompt filtering, RAG data protection, and seamless framework integration with popular AI development tools.

Identity and Lifecycle Management for Agents

Identity Lifecycle Management (ILM) has evolved to address the unique challenges of AI agents and automated systems. Traditional ILM focuses on human identities, but agentic AI requires management of dynamic, autonomous, ephemeral actors.

Modern agent identity management includes:

• Agent authentication through cryptographic proofs using SPIFFE/SVID, PKCE, and mTLS + JWT tokens

• Dynamic identity issuance with ephemeral, just-in-time credentials

• Lifecycle governance through automated provisioning and deprovisioning

• Delegation tracking from user to agent to downstream services

  
  

Organizations are implementing automated user provisioning and deprovisioning systems that can swiftly revoke access and delete user accounts with minimal time and effort. This prevents scenarios where former employees retain access to critical applications after leaving.

Input Validation and Output Filtering

Input validation and output filtering represent critical first-line defenses against API injection attacks and data manipulation. These techniques are essential for preventing common vulnerabilities like SQL injection, cross-site scripting (XSS), and command injection.

Best practices include:

• Schema validation to reject malformed data before it reaches business logic

• Type checking and data coercion with explicit validation and clear error messages

• Size and range validation to prevent denial-of-service attacks

• Allowlisting approaches that define authorized inputs and reject everything else

• Multi-agent frameworks for prompt injection detection and mitigation

  
  

For AI systems specifically, advanced data sanitization processes eliminate potentially harmful data, ensure consistency, and safeguard sensitive information from unauthorized access. Modern solutions like Wald.ai provide real-time data sanitization during live interactions with AI assistants.

Real-Time Anomaly Detection

Real-time anomaly detection has become essential for identifying sophisticated bot behaviors that often go unnoticed by conventional techniques. Modern systems combine multiple detection methods to identify deviations from established behavioral patterns.

Advanced implementations include:

• Time-series anomaly detection that analyzes request patterns over specific intervals

• Dynamic threshold management that recalculates optimal thresholds based on real-time feedback

• Behavioral analytics using User and Entity Behavior Analytics (UEBA) to monitor both human and non-human entities

• Machine learning models that build dynamic behavioral profiles from legitimate traffic

Leading providers like Radware implement isotonic calibrators that transform raw anomaly scores into probability-based scores, enabling more accurate threat assessment. These systems can detect sophisticated attacks like low-and-slow campaigns that spread activities across time to avoid typical rapid-fire detection patterns.

Audit Logging and Visibility

Comprehensive audit logging for AI agents and bots has become essential for security, compliance, and operational integrity. Modern audit systems capture detailed information about agent operations, decision-making processes, and interactions.

Key components include:

• Comprehensive logging of all compliance-related activities with tamper-evident trails

• Intelligent search capabilities for quickly analyzing vast amounts of data during investigations

• Pattern recognition to identify hidden compliance issues and connections

• Real-time monitoring with automated alerts for suspicious activities

Microsoft's audit logs for Copilot and AI applications demonstrate enterprise-grade implementation, automatically logging user interactions and admin activities as part of standard audit processes. These systems provide detailed visibility into AI operations while maintaining privacy and security standards.

Human-in-the-Loop for Critical Actions

Human-in-the-loop (HITL) processes provide essential oversight for AI agents, but research reveals significant limitations in human oversight effectiveness. While HITL is crucial for high-risk scenarios, implementation must account for human cognitive limitations.

Effective HITL implementation requires:

• Delegated final decisions to humans for sensitive transactions

• Real-time communication between agents and humans using WebSocket connections

• Structured approval workflows with clear criteria and escalation procedures

• Multi-step transparent workflows that allow users to understand AI reasoning

However, research indicates that humans often cannot accurately assess the quality of algorithmic recommendations and commonly fail to correct harmful AI decisions. The EU AI Act's reliance on human supervision may be problematic, as mounting evidence shows such oversight is not always reliable.

Sandboxing and Safe Release Strategies

Sandboxing provides isolated, controlled environments for testing AI agents without harming critical systems. This approach has become standard practice, with 93% of Salesforce customers considering sandboxes essential for AI-driven development.

Modern sandboxing approaches include:

• Container-based isolation using technologies like Docker with namespace isolation and cgroups

• Virtual machine environments with hardware-level isolation for security

• Dedicated testing environments that mirror production data and configurations

• Progressive deployment strategies moving through development, testing, and production phases

Platforms like E2B provide open-source runtime for executing AI-generated code in secure cloud sandboxes, specifically designed for agentic AI use cases. These systems enable safe experimentation and development without risking production environments.

Advanced Fingerprinting and Signature Verification

Advanced fingerprinting techniques have become sophisticated tools for bot detection and identity verification. Modern systems combine device attributes, network information, and behavioral patterns to create unique profiles.

Current implementations include:

• Device fingerprinting using browser characteristics, screen resolution, installed fonts, and hardware details

• Behavioral biometrics analyzing mouse movements, keystroke dynamics, and interaction patterns

• Multi-modal detection combining various signals for improved accuracy

• Machine learning enhancement for identifying automation tools and frameworks

Leading solutions like Fingerprint Pro can detect sophisticated automation tools including Selenium, Playwright, and stealth plugins while distinguishing between good and bad bots. These systems achieve high efficacy with near-zero false positives while maintaining access for legitimate crawlers.

Robots.txt and Custom Policies: Understanding Limitations

While robots.txt remains a widely-used standard, research reveals substantial limitations in its effectiveness for bot control. The protocol is purely informational and cannot enforce restrictions on malicious crawlers.

Key limitations include:

• Voluntary compliance only - malicious actors have zero incentive to abide by robots.txt directives

• No enforcement mechanism - bots can simply ignore the file without consequences

• High non-compliance rates - studies show 56.25% of bots ignore robots.txt directives

• Security risks - the file can reveal sensitive directory structures to attackers

However, modern solutions like Cloudflare's AI Audit allow organizations to enforce robots.txt policies through Web Application Firewall rules, providing actual blocking capabilities rather than relying on voluntary compliance. Custom bot management policies enable more sophisticated control through managed rules and custom matching conditions.

All suggested security strategies have been validated as effective and necessary components of a comprehensive bot and AI agent security framework. However, implementation must account for the specific limitations and evolving nature of each approach. The most effective strategy combines multiple validated techniques in a layered defense architecture, recognizing that no single solution provides complete protection against sophisticated automated threats.

Organizations should prioritize implementing these strategies based on their specific risk profiles and operational requirements, with particular attention to the limitations of voluntary compliance mechanisms like robots.txt and the cognitive limitations of human oversight systems.

Looking Ahead

The future of web security depends on visibility, control, and adaptability. As bots and AI agents become the primary consumers of web content, organizations must move beyond blunt blocking and embrace context-aware, layered defenses. The ability to distinguish between good and bad automation, adapt policies in real time, and empower both users and agents to interact safely will define the next era of online security.

A growing ecosystem of companies is targeting this specific niche. In addition to Arcjet, leaders like Cloudflare, HUMAN Security, Akamai, Radware, DataDome, Zenity, and 7AI are all innovating in bot and agent management. These companies leverage advanced machine learning, behavioral analysis, device fingerprinting, and real-time decision engines to help organizations distinguish between helpful and harmful automation, protect APIs and applications, and ensure a trusted digital environment.

The web is no longer just for humans, and the strategies for protecting it must evolve accordingly.