Zero trust and AI security: securing identity, data and autonomous systems

Summary:

• Zero trust shifts security from the network perimeter to the point of access, requiring every request to be verified before resources are granted
• Identity-driven security combines authentication, entitlements and contextual risk to determine who can access data and services
• AI systems should be treated as constrained workloads with limited permissions, not trusted internal users with broad access
• Continuous verification, least-privilege access and independent policy enforcement are essential for secure AI adoption

As organisations continue to expand across cloud environments, third-party platforms and AI-powered services, the traditional concept of trust based on network location is becoming increasingly obsolete. Security is no longer defined by whether a user or system operates within a corporate boundary, but by the ability to verify every access request, validate every interaction, and apply controls that are proportionate to the risk.

This shift has accelerated the adoption of zero trust principles, which place identity, context, entitlement and continuous verification at the centre of modern security strategy.

At the same time, the emergence of generative AI introduces new challenges around autonomy, data access and decision making, requiring organisations to rethink how established security concepts are applied in an increasingly intelligent and interconnected technology landscape.

How should organisations define the core boundary of a zero trust architecture today?

The core boundary should now be defined around each protected resource and each request made to that resource, rather than around the corporate network. In my work at Fidelity, I have supported the modernisation of four related trading and wealth-management application workloads involving Angular interfaces, Spring Boot services, AWS environments and several downstream integrations. Although these components operated within approved enterprise environments, we could not assume that every connection was automatically trustworthy.

We mapped application flows, established a reusable firewall and connectivity-onboarding process, secured token propagation and limited communication to approved sources, destinations and services. Access also had to respect the user’s identity and entitlements. That experience reinforced for me that the real zero trust boundary is the point at which a user, workload, or service requests access to an API, application function, or dataset. Network segmentation remains valuable for reducing exposure, but it should contain risk rather than serve as evidence of trust.

In an era of hybrid work, SaaS sprawl and BYOD, how fundamentally has the definition of the network edge shifted towards identity-driven security?

The shift has been fundamental. The edge is no longer a physical location; it is wherever an access decision must be made. However, identity-driven security should mean more than authenticating someone once through an IAM platform. The identity of the user or workload must remain associated with the request as it moves through APIs, servicesnand cloud environments, and it must be evaluated alongside entitlements, device condition, session context and the sensitivity of the requested operation.

During AWS migration and integration work at Fidelity, we combined identity controls with narrowly defined network connectivity. Firewall approval covered the precise communication path, while the application and service layers still validated the caller and the permitted action. A corporate device or internal IP address was not sufficient justification for unrestricted API access. In practical terms, identity establishes who is requesting access, entitlements establish what they may do, network controls limit how they can reach the resource, and ongoing telemetry helps determine whether that access should continue.

From a zero trust perspective, why is treating an AI model or pipeline as a trusted internal user potentially dangerous?

An AI model should not be treated as a trusted internal user because it is not simply reading information; it is interpreting instructions from users, as well as retrieved documents and connected systems. Any of those inputs may be incorrect, manipulated or malicious. If the model operates through a broadly privileged service account, it can unintentionally combine access that no individual user should possess. A harmful instruction hidden in a document could then influence the model to retrieve sensitive information or call an authorised tool for an unauthorised purpose.

My direct production experience has been with secure application and API integrations rather than autonomous AI agents, but the same principle applies. In our financial services applications, security tokens were propagated through controlled mechanisms and were not exposed through user interfaces, URLs or application logs. Downstream services were still responsible for authentication and authorisation. For AI, I would apply dedicated workload identities, short-lived, narrowly scoped tokens, user-specific entitlement checks, and independent validation of every tool call. The model can propose an action, but it should never be allowed to decide on its own. 

How do you apply least-privilege access and continuous verification to software that makes its own decisions?

An autonomous agent should be treated as a constrained workload, not as a digital employee with a permanent set of broad permissions. Each agent and use case should have a separate workload identity. Access should be read-only by default, credentials should be short-lived, and the agent should access enterprise systems via approved APIs or controlled procedures rather than unrestricted database credentials. Every tool call should be independently authorised using the user identity, agent identity, requested operation and current risk context.

There is a practical parallel with the cloud applications I worked on at Fidelity. Connectivity was limited to the required application flows, entitlements were defined for specific functions, and production changes went through controlled testing and approval stages. Autonomy should not remove those controls. For higher-risk AI actions, such as changing financial data, deleting records or initiating external communications, I would require human confirmation or approval from a second deterministic service. Immutable audit records, transaction limits, anomaly monitoring and an immediate method of revoking the agent’s access are also essential. Continuous verification means the agent earns permission for each material action, not once at the beginning of a session.

How can security teams maintain visibility and integrity across the entire AI data supply chain?

Visibility begins with data lineage. An organisation should be able to identify where every important dataset originated, who owns it, how it is classified, what transformations were applied, which model or retrieval system uses it and where the resulting information is stored. Raw data, curated training data, retrieval indexes and evaluation datasets should be separated, versioned and protected according to their purpose. Integrity checks, data-quality gates, encryption, access logging and monitoring for unusual changes should be part of the normal pipeline.

This is similar to an important lesson from cloud migration work. Before moving an application, we had to understand its upstream and downstream dependencies, communication paths and production behaviour. Unknown dependencies could otherwise become operational or security gaps after deployment.

For AI, the same discipline must extend beyond databases to vector stores, embeddings, prompts, model versions, external datasets and third-party components. Dynamic data should be checked against the user’s current entitlements when it is retrieved; organisations should not copy an entire sensitive database into an AI-accessible store merely because it is technically convenient.

What does a zero trust runtime defence look like when the attack vector is natural language?

Natural language supplied to an AI system must be treated as untrusted input, even when it comes from a document, email or website that appears legitimate. A system prompt alone is not a security boundary because a model may still be influenced by conflicting instructions embedded in retrieved content. Runtime defence therefore needs several independent layers.

For you

Be part of something bigger, join BCS, The Chartered Institute for IT.

Sensitive credentials should never be placed in the model’s context. Retrieved content should retain its source and trust classification, model outputs should follow constrained formats, and every requested action should pass through an external policy-enforcement service. Tool access should be allow-listed, data leaving the environment should be inspected, and higher-risk operations should require user confirmation.

The underlying principle is familiar from secure API development: a backend service does not approve an operation merely because the request came through an approved user interface. It independently validates the input, identity, entitlement and requested action. With AI, the model may interpret or recommend, but a deterministic control outside the model must decide what is permitted.

What does a realistic roadmap look like for retrofitting zero trust principles into an organisation that has already rushed to adopt generative AI?

The first step is containment and visibility, not immediate replacement. During the first phase, the organisation should inventory every generative AI use case, model, dataset, vector store, plugin, API connection and service account. It should then rank them according to the sensitivity of the accessible data and the consequences of the actions they can perform. Shared credentials and unnecessarily broad permissions should be addressed first.

The next phase is to introduce dedicated workload identities, short-lived tokens, centralised policy enforcement, and a clear separation between read-only assistance and actions that change business data. Logging should capture retrieval activity and tool calls without unnecessarily recording sensitive prompt content. The organisation should then test for prompt injection, data leakage, privilege escalation and failure of connected services before expanding the deployment.

This staged approach reflects how I have handled enterprise cloud modernisation. We mapped dependencies, created repeatable connectivity controls, validated changes in non-production environments, addressed vulnerabilities and moved through controlled production installations. I contributed to eight production installations over approximately ten months using that disciplined progression.

Retrofitting AI security should follow the same pattern: control the highest-consequence connections first, prove the approach and then scale it across the organisation.

Looking at the next three to five years, do you see AI as a net positive or negative technology for professionals working in security? Does it advantage defenders or attackers?

My answer is clear: over the next three to five years, AI will be a net positive for defenders. Attackers will certainly use it to increase the speed and scale of phishing, reconnaissance, vulnerability discovery and social engineering. However, defenders possess something attackers generally lack: detailed knowledge of their own identities, application dependencies, system behaviour, security telemetry and authorised business processes.

AI can help security teams correlate large volumes of events, identify abnormal behaviour, review application and infrastructure code, detect configuration drift and prioritise vulnerabilities much faster than manual processes alone. In my own work, keeping four related application workloads current, addressing vulnerabilities, and supporting repeated production deliveries required disciplined, often time-consuming analysis. AI can reduce that burden and allow experienced professionals to focus on architecture, risk decisions and remediation. It must still operate within scoped permissions and human accountability, but defenders have both the context and authority to correct the weaknesses AI identifies. Attackers may gain volume; properly prepared defenders will gain context, speed and precision.