Why AI agents need software architecture, not just better prompts

When an AI agent produces the wrong result, the first reaction is often to rewrite the prompt. But prompt engineering cannot compensate for a poorly designed system.

Once an AI application starts retrieving enterprise data, calling APIs, or executing business actions, it has become a software system. The language model may provide reasoning, but the surrounding architecture determines whether that reasoning can be used safely and reliably.

AI agents are more than LLMs

A basic AI application receives a request, sends it to a model, and returns a response. An AI agent does more: it decides which tool to call, retrieves information across systems, maintains state, and determines when human approval is required.

A production-grade agent includes an orchestration layer, access controls, retrieval mechanisms, validation, observability, and approval workflows. A better prompt will not solve authorization, auditability, or uncontrolled execution loops – these are architectural concerns.

The model should not enforce business rules

Here’s a principle worth applying: don’t place critical responsibility inside the model. An LLM can interpret intent and recommend an action, but it should not be the only component deciding whether that action is permitted.  

Consider an agent that can approve a claim, update an employee record, deploy software, or send an external email. A prompt instruction like “only perform this action when authorized” is not a security control. Authorization must be enforced by the application executing the action, which independently verifies identity and scope. 

Prompts guide behavior. Architecture enforces behavior.

AI agent tools need clear boundaries

Agents become valuable interacting with real systems such as databases, but unrestricted access creates unnecessary risk.  

Every tool to an agent should have a clear contract: what it does, what permissions it requires, and whether it’s reversible. A broad tool like execute_database_query gives the model a large execution surface; narrower tools like get_customer_claims are easier to secure and audit. 

The same applies to protocols like the Model Context Protocol; it standardizes how tools are exposed, but doesn’t solve authentication or tenant isolation on its own.  

The protocol defines connectivity. The architecture defines control. 

Enterprise retrieval for AI agents must preserve access control

Many AI applications rely on Retrieval-Augmented Generation (RAG), but retrieving the right information is only half the problem. The harder challenge is ensuring the user is allowed to see it. Source-system permissions must be preserved, data must remain isolated across tenants, and every answer should trace back to its source.  

For sensitive repositories such as email, contracts, healthcare records, or financial systems, additional controls including data classification, PII detection, secret detection, masking, and tokenization may also be required.

A vector database should not become a second, less-controlled copy of company data. Asking the model not to reveal sensitive data is not enough. Simply asking the model not to reveal confidential information is not a security strategy. 

Agentic loops need limits

Much of what makes agents feel intelligent comes from their ability to iterate. They interpret a goal, choose an action, call a tool, evaluate the result, and repeat until they believe the objective has been achieved. 

That same powerful capability also introduces unpredictability. Without safeguards, an agent may repeatedly invoke a failing tool, exceed cost limits, or never reach a stopping condition. Production systems should define explicit limits for iteration counts, timeouts, retry policies, token budgets, idempotency controls, and stopping conditions. 

Not every process should be fully autonomous. Often a controlled workflow with AI-assisted steps is safer: retrieve, analyze, validate, apply business rules, request approval, then execute through a controlled service, letting the model reason without controlling the workflow. 

Structured output is an interface

Whenever another system consumes model output, free-form text becomes a liability. An agent reviewing code should return a defined structure, severity, file path, finding, remediation, which becomes the interface between model and application. 

Asking the model to return JSON is only the beginning. The application must still verify the output is valid. Model output should be treated like input from any external dependency: useful but untrusted until validated. 

Observability must explain agent behavior

AI agents need the same monitoring as traditional systems, plus visibility into their decisions.

Teams need to know which model and prompt version were used, what information was retrieved, which tools were called, why validation failed, how many steps the task required, how much time and cost were consumed, and whether a human accepted or corrected the result. Without this, debugging becomes guesswork; the problem may not be the prompt, but poor retrieval, stale data, or an orchestration decision.

Good observability turns AI behavior from a black box into an engineering problem that can be diagnosed.

Evaluation goes beyond accuracy

Because AI systems are probabilistic, they cannot be evaluated with traditional pass-or-fail testing alone.

A production agent should be assessed on several dimensions, such as factual correctness, tool-selection accuracy, schema compliance, security-policy compliance, and cost, including adversarial cases like prompt-injection attempts.

The goal is not to prove that the model is always correct. It is to understand where it is reliable, where it fails, and what controls are needed around those failures.

Human approval should match the risk

Oversight isn’t a generic checkbox; it should match the impact of the action. Reading information needs little supervision; changing production data or approving payments needs stronger controls. The architecture decides what’s automatic, what needs confirmation, and what should never be delegated to an agent. Approval screens should also show evidence, affected systems, and proposed changes, otherwise the reviewer is simply approving blind.

Production AI is still software engineering

As language models become more capable, organizations connect them to more valuable systems and give them greater responsibility, which makes engineering discipline more important, not less.

Production AI still requires secure system boundaries; resilient workflows, testable interfaces, least privilege access, observability; versioning, change management, and failure recovery. A well-written prompt can improve an agent’s response. A well-designed architecture determines whether it can be trusted with real work.

The organizations that succeed with agentic AI won’t simply be those with the most advanced models or cleverest prompts. They’ll be the ones that treat AI agents as production software systems and apply the same engineering discipline expected from any business-critical platform.

The model provides intelligence. The architecture makes that intelligence usable.

Elie Saliba

Senior Software Engineer