Secure AI Engineering

Building Secure AI Agents: Architecture, Identity, Tools and Guardrails

With new agent SDKs from the major model providers, MCP support spreading and Google's Agent2Agent protocol just announced, building agents has never been easier. This is a builder's guide to doing it securely: the architecture, identity model, tool design and guardrails that should be in place before an agent touches real systems.

By EthicalHat Research · · 3 min read
Infographic: A secure agent, layer by layer

Building an AI agent has become dramatically easier this spring. OpenAI released its Agents SDK in March, Google introduced its Agent Development Kit this month, and open-source frameworks keep maturing. MCP gives agents a standard way to reach tools, and Google's Agent2Agent (A2A) protocol, announced two weeks ago, aims to let agents from different vendors work together.

The ease of building agents makes it all the more important to build them securely. This guide is for engineers and architects designing agents that will touch real data and systems.

Architecture: a deterministic shell around a probabilistic core

The model at the centre of an agent is powerful, fallible and manipulable. Wrap it in a layer of deterministic code that it cannot talk its way past:

flowchart TB
  In[Inputs: user, events,<br/>documents, other agents] --> IG[Input guardrails<br/>limits, classification,<br/>untrusted-content marking]
  IG --> LLM[Model: plan & propose]
  LLM --> OG[Output validation<br/>schemas, allow-lists]
  OG --> PZ{Policy check<br/>identity, scope, data, limits}
  PZ -->|allow| EX[Executor<br/>sandboxed, idempotent]
  PZ -->|approval| H[Human approval]
  PZ -->|deny| D[Deny + explain]
  H --> EX
  EX --> Tools[Tools / MCP servers]
  EX --> T[(Trace & audit)]
  Tools --> LLM

The model decides what it would like to do. The shell decides what actually happens.

Identity

  • Every agent gets its own identity: a registered, non-human principal with an owner.
  • Agents act on behalf of someone: carry the delegating user's identity with every action, and enforce the intersection of the user's permissions and the agent's scope.
  • Short-lived, scoped credentials: obtain tokens per task (for example via OAuth token exchange or your cloud provider's workload identity), never long-lived shared keys.
  • Agent-to-agent calls are authenticated. With A2A and similar protocols, verify which agent is calling and what it is allowed to ask for; do not treat messages from other agents as trusted instructions.

Tools

Tools are where agents create impact. Design them like security-critical APIs:

  • Narrow: refund_order(order_id, amount) rather than execute_sql(query).
  • Strictly typed: schemas with enums, ranges, patterns and required fields.
  • Validated server-side: never trust arguments because the model produced them.
  • Idempotent: accept idempotency keys, because agents retry.
  • Least privilege: each tool's credentials allow only its job.
  • Clearly described: short, factual descriptions (they are part of the prompt).
  • Classified by risk: read-only, reversible write and irreversible write, with approval requirements attached.
python
from pydantic import BaseModel, Field, constr
from decimal import Decimal

class RefundArgs(BaseModel):
    order_id: constr(pattern=r"^ORD-\d{6}$")
    amount: Decimal = Field(gt=0, le=Decimal("500"))
    reason: constr(max_length=200)
    idempotency_key: constr(min_length=8, max_length=64)

@tool(risk="irreversible", requires_approval_over=Decimal("50"))
def refund_order(args: RefundArgs, ctx: AgentContext):
    order = orders.get(args.order_id, customer_id=ctx.on_behalf_of.customer_id)  # scoped
    if args.amount > order.refundable_amount:
        raise ToolError("amount exceeds refundable balance", refundable=str(order.refundable_amount))
    return payments.refund(order, args.amount, idempotency_key=args.idempotency_key)

Guardrails

  • Input: size limits, abuse detection, and marking content from documents, web pages and other agents as untrusted.
  • Output: schema validation, allow-listed destinations and encoding before rendering.
  • Action: policy checks on every tool call; approvals for high-impact actions with an exact, code-generated summary.
  • Budget: limits on steps, time, tokens and spend per task.
  • Data flow: rules that prevent sensitive data reaching outbound channels.

Isolation

Agents that browse, run code or manipulate files should do so in sandboxes (containers or microVMs) with no ambient credentials and restricted network egress. Keep memory stores isolated per user or tenant.

Observability

Trace every run end to end: inputs, retrieved context, model calls, proposed actions, policy decisions, approvals, executions and results. Traces are how you debug, evaluate, audit and investigate incidents. Without them, an agent is a black box with privileges.

Testing before launch

  • Functional evals on real tasks.
  • Adversarial tests: prompt injection through every input channel, tool misuse attempts, cross-user access and data exfiltration scenarios.
  • Failure tests: tool errors, timeouts and partial failures, to confirm the agent fails safely.

A pre-launch checklist

  • Agent registered with an owner, purpose and allowed tools
  • Own identity; acts with delegated, least-privilege scope
  • Every tool narrow, typed, validated and idempotent
  • Policy check on every action; approvals for high-impact actions
  • Untrusted content marked; outbound data flows controlled
  • Sandboxed execution; no ambient credentials
  • Budgets on steps, time and cost
  • End-to-end tracing and audit logging
  • Adversarial test suite passing, and running in CI

The bottom line

Secure agents are built, not prompted. Put a deterministic shell of identity, authorisation, validation, isolation and tracing around the model, design tools as if an attacker will call them, and test adversarially before launch. The new SDKs and protocols make agents easy to build; these practices make them safe to run.

Our agentic AI security team builds agents on this architecture, and our application security testing verifies them before they go live.

EthicalHat Research

The EthicalHat research team: penetration testers, SOC analysts and engineers writing about security, software engineering and AI.

Want an expert opinion on your own systems?

Email info@ethicalhat.com or call +1-844-838-4422. Our security engineers are happy to talk it through.