Agent Architectures: How to Design Reliable AI Agents

Maya asks an internal assistant, “Why did yesterday’s payment captures fail, and what should the on-call team do?”

The assistant needs to search logs, check the payment provider’s status, read the team runbook, and write a short incident summary. It could make one model call with every tool attached. It could also split the job among a planner, a log investigator, a runbook reader, and a reviewer.

Both options can work. The second is not automatically more capable. More agents mean more model calls, more state to pass around, more handoffs to debug, and more ways for a confident-looking mistake to spread.

Agent architecture in plain English: it is the arrangement of model calls, tools, state, and rules that decides how an AI system gets work done.

This article starts with the simplest arrangement and adds structure only when the simpler one fails. That is the useful way to think about agents: not as a collection of named bots, but as a controlled way to complete a task.

1. First, separate a workflow from an agent

An AI workflow has a path chosen by code. An agent lets the model choose at least part of that path: which tool to use next, whether it has enough evidence, or when to ask for help.

Workflow:  classify → retrieve policy → draft reply → send for approval
             (the application chose every step)

Agent:     inspect request → choose a tool → observe result → choose next step
             (the model makes bounded decisions)

Neither label tells you whether a system is good. A workflow is usually better when the work has a known sequence and mistakes are expensive. An agent earns its extra freedom when the path genuinely varies: a coding task may touch an unknown number of files; an investigation may require a different next query after every result.

This distinction is also how Anthropic describes the terms: workflows use predefined code paths, while agents dynamically direct their own tool use. Their practical advice is sound: begin with the simplest solution and add complexity only when it improves task performance. Building Effective Agents

For Maya’s request, the first question is not “How many agents?” It is “What choices must the system make, and which of them are safe to let a model make?”

2. The smallest useful architecture

An agent is more than an LLM with a dramatic job title. The minimum useful shape has five parts:

User goal
   ↓
Instructions + current state
   ↓
Model ── chooses ──→ tool or final answer
   ↑                    ↓
   └────── tool result / observation

Controls around the loop: permissions, budgets, approval, stop conditions, logs

• Instructions define the job and its boundaries: “Investigate capture failures; never restart a service.”
• Tools retrieve facts or take actions: search logs, read a runbook, open an incident.
• State holds the useful working record: the user’s request, tool results, and decisions already made.
• The loop gives the model a chance to react to what it learns rather than guessing everything upfront.
• Controls cap the freedom: a maximum number of tool calls, a token budget, timeouts, user approval, and an escalation path.

The loop is what makes it agentic:

while not done and within_budget:
    model chooses the next safe step
    application validates and runs the requested tool
    result becomes the next observation

The model may decide the next step. It should never become the final authority for permissions. As with Model Context Protocol, trusted application code must validate every tool call and enforce authorization.

3. Start with a deterministic workflow

Maya’s team may discover that every payment-capture incident needs the same three facts: error counts, provider status, and the relevant runbook section. There is no prize for making the model decide that sequence.

Request
   ↓
Query payment-error dashboard
   ↓
Check provider status page
   ↓
Retrieve payment-failure runbook
   ↓
LLM writes a cited summary
   ↓
Human on-call engineer decides what to do

This prompt chain or sequential workflow is easy to test. Each step has a small input and output; the application can retry a failed status lookup without repeating the whole task. It is a strong default for document transformations, known approval flows, and reports built from a fixed set of sources.

Its limitation is equally clear. If the dashboard shows an unfamiliar error code, the fixed chain cannot decide whether to query a deployment log, inspect a specific merchant, or stop and ask Maya for more detail. That is the point at which a loop may help.

4. The single-agent loop: the default next step

Before building a team of agents, give one agent clear tools and a clear stopping rule.

Maya's request
      ↓
Investigation agent
      ├── search_payment_errors(error, time_window)
      ├── get_provider_status()
      ├── read_runbook(topic)
      └── prepare_incident_summary(evidence)
      ↓
Final answer, escalation, or approval request

The agent can search errors first, discover that failures began after a deployment, then read the matching runbook. That flexibility is useful because the next useful question depends on the previous answer.

Give the loop explicit exits. For example:

Finish        Enough evidence for a cited summary
Escalate      Conflicting evidence, missing access, or a high-risk action
Stop          8 tool calls, 90 seconds, or $0.25 of model budget
Fail safely   Tool timeout or invalid result; do not invent a conclusion

One agent is easier to evaluate, trace, and improve than a network of specialists. OpenAI similarly recommends starting with a single agent and adding tools incrementally before moving to multi-agent orchestration. A practical guide to building agents

5. Add a router when the task has stable categories

Some requests belong to clearly different lanes. A support system might receive “Where is my order?”, “I need a refund,” and “The app crashes.” The tools, policies, and risk levels differ.

Incoming request
      ↓
Router
  ┌───┼───────────┐
  ↓   ↓           ↓
Orders workflow  Refund workflow  Technical-support workflow

A router can be ordinary code, rules, a small classifier, or an LLM. Choose the simplest classifier that meets the error rate you can tolerate. A request with an order number may not need a model to identify it as an order question.

Routing is useful when specialization makes prompts and tool permissions clearer. It is not useful when the categories are fuzzy and every handoff loses context. In that case, one agent with good instructions often behaves better than a router that guesses wrong.

6. Run independent work in parallel

Parallelism is not a multi-agent personality trait. It is a latency decision.

For Maya’s incident, querying the error dashboard and provider status are independent. Start both at once, then ask one model call to combine the evidence:

                    ┌── query error dashboard ──┐
Request ── fan out ─┼── check provider status ───┼── synthesize evidence
                    └── retrieve runbook ───────┘

Parallel work helps in two situations:

• independent retrievals where waiting for one before starting another adds no value; and
• independent reviews where several perspectives improve confidence, such as a security review plus a correctness review.

It also creates a merge problem. Decide ahead of time what happens when one result fails or the reviewers disagree. “Ask another agent to decide” is often just postponing the design. For high-stakes decisions, define a deterministic policy or route the disagreement to a human.

7. Use planner–executor only when the plan is genuinely uncertain

In a planner–executor design, one model proposes a plan and another execution loop carries it out. The executor returns observations; the planner can revise the plan.

Goal: explain a capture failure
      ↓
Planner: 1. inspect errors  2. compare deploys  3. read runbook
      ↓
Executor: run step 1 → result
      ↓
Planner: error began after deploy 842; revise step 2
      ↓
Executor: inspect deploy 842 → result

This pattern earns its cost when a task spans many dependent steps and early observations change the later plan: a code migration, deep research, or a complex operational investigation.

Avoid it for a three-step task with known inputs. A written plan can become ceremony: it consumes context, gives the system another output to validate, and may anchor the executor to a bad first idea. A single loop can often plan informally and act just as well.

8. Multi-agent architectures: two useful shapes

Multiple agents are a way to separate responsibilities, not a badge of sophistication. Add them when one agent’s tool list or instructions have become so broad that it chooses poorly, or when different domains need different permissions and evaluation criteria.

Manager and workers

A manager owns the user-facing task. It delegates bounded subtasks to workers and combines their results.

                    Manager
               /       |       \
              ↓        ↓        ↓
         Logs worker  Docs worker  Deployment worker
              \        |        /
                 evidence bundle
                      ↓
                 Manager's answer

The manager remains accountable for the final answer. Workers should return a compact, structured evidence bundle rather than a long private conversation. This is a good fit when subtasks can happen independently and the user should receive one coherent answer.

Handoffs between peers

In a handoff design, an agent transfers the active conversation to a specialist. A support triage agent may hand a cancellation request to a cancellation agent, which then owns the rest of the interaction.

Triage agent ── handoff ──→ Cancellation agent ──→ user

Handoffs fit a conversational experience where one specialist should take ownership. Define the transfer contract: what context moves, who can make which tools available, and who can hand the conversation back. Without that contract, the user gets bounced between bots and no one owns the result.

Manager-as-tools and peer handoffs are the two broad multi-agent patterns described in OpenAI’s practical guide. Orchestration patterns

9. State and memory: keep a record, not a diary

Every architecture needs state. The common failure is calling all of it “memory.” Separate it by purpose:

Run state        Evidence and decisions for this request; expires with the run
User memory      Stable preferences the user has approved; changes slowly
Knowledge        Documents and records retrieved when needed; source of truth lives elsewhere
Audit trail      Tool calls, approvals, and outcomes; retained for operations and compliance

Do not keep appending the entire conversation to every model call. It gets expensive, buries the relevant facts, and makes it harder to tell which evidence supported a decision. Store durable facts in ordinary systems of record. Retrieve the specific facts the next step needs, and pass a short structured summary of prior work.

For Maya’s incident, the run state might be: error time window, relevant deployment ID, provider status, cited runbook steps, and tool-call IDs. That is enough to resume after a timeout without pretending the agent has perfect long-term memory.

10. Reliability comes from controls and evaluation

An architecture diagram without failure handling is a demo diagram. Agents can call the wrong tool, retry a side effect, loop forever, misread a tool result, or follow hostile instructions embedded in retrieved text.

Design for those cases at the edges of the loop:

Before a tool call   Validate arguments and authorization
During execution     Timeout, retry read-only calls, use idempotency keys for writes
After a tool result  Check schema and freshness; preserve provenance
Before a write       Require user approval when the consequence warrants it
At every step        Enforce step, time, token, and cost budgets
At the end           Emit a trace with decisions, tool calls, and outcome

Then evaluate the actual task, not whether the model sounds confident. Build a small set of representative requests and score outcomes such as:

• Did the agent reach the correct resolution or escalate appropriately?
• Did it use only allowed tools and permissions?
• Did a retry create one side effect or two?
• Did the final answer cite the evidence it actually retrieved?
• How many tool calls, seconds, and tokens did the run consume?

Traces are especially useful. They show the model’s decisions, the tool inputs and results, and where the run stopped. Without them, “the agent failed” is not actionable. Guardrails, observability, and human intervention are core production components, not polishing after the architecture is done. OpenAI’s guidance on guardrails and intervention

11. Choosing an architecture

Use the smallest design that gives the needed reliability and flexibility.

Known, fixed steps?                         Deterministic workflow
One changing path with a few tools?         Single agent with limits
Stable request categories?                  Router + focused workflow/agent
Independent retrieval or review?            Parallel tasks + explicit merge
Long task where findings change the plan?   Planner–executor
Distinct domains or permissions?            Manager-workers or handoffs

The progression is intentional. A multi-agent system can be the right answer, but it is usually the last answer. First make tools clear, reduce the tool surface, improve instructions, add an evaluation set, and see whether a single agent still fails for a repeatable reason.

12. The mental model to keep

An agent architecture is not a set of prompts connected by arrows. It is a control system around an LLM.

Model          Chooses among bounded next steps
Tools          Observe or act on the outside world
State          Carries only the facts needed for the next decision
Architecture   Decides what can run in sequence, parallel, or by delegation
Controls       Limit cost, time, permissions, and unsafe actions
Evaluation     Shows whether the system completes real tasks reliably

Start with the workflow you can explain on one whiteboard. Give a single agent a narrow, well-tested tool set when the next step must adapt to evidence. Add routing, parallel work, planning, or multiple agents only when you can name the failure they solve. That restraint is what makes an agent system understandable, operable, and safe.