Amit Lulla recently shared something that caught my attention: "AI agents creating their own social networks. Agents developing communication systems humans can't understand. Autonomous exploration of abstract concepts." He followed it with the line that should make every builder pause: "The question everyone's asking: 'Who's accountable when these systems behave unexpectedly?' Right now? Nobody. And that's the problem."

I have seen the same pattern across agentic AI projects: teams get multi-agent demos working, ship quickly, and only later ask what happens when the system surprises them. Amit's point is not sci-fi fearmongering. It is a production engineering reality.

In this post, I want to expand on what Amit is really saying: agentic systems do not fail like traditional software. They fail like complex organizations. Small permission mistakes, weak boundaries, and missing observability can compound into large, costly incidents.

The real risk is not power, it is missing rigor

Amit wrote: "The risk isn't that agents are 'too powerful.' The risk is deploying systems with agency without production rigor." That framing matters.

When you give a system agency, you are doing at least three things at once:

You are delegating decisions (not just computations).
You are enabling actions through tools (APIs, databases, email, browsers, payment rails).
You are allowing feedback loops (agents react to outcomes and try again, sometimes repeatedly).

Traditional apps can crash or return wrong answers. Agentic systems can keep going, try alternatives, and create new plans. That persistence is useful, but it changes your safety and governance requirements.

Key insight: Agency turns bugs into behaviors.

Why accountability is fuzzy in multi-agent systems

Amit's accountability question lands because many agentic stacks blur responsibility across layers:

The model decides what to do.
The orchestrator decides which agent runs.
A tool router decides which API gets called.
External services and data sources shape the outcome.

When something goes wrong, teams often cannot answer basic questions:

Which agent initiated the action?
What was the exact prompt and context at the time?
Which tool call changed the world state?
Was there a human approval step available but bypassed?

If you cannot answer those quickly, you do not have accountability, even if someone is "on call."

What "zero governance architecture" looks like in practice

Amit said most multi-agent systems ship with "no audit trails. No circuit breakers. No observable decision paths." Here are common failure modes that sit behind that statement.

1) Invisible decisioning

Without structured logs, you get a pile of unsearchable text traces. That is not an audit trail. An audit trail is a chain of events that can be replayed and reviewed.

Minimum viable auditability usually includes:

A unique run ID per workflow and per agent turn
The tool name, parameters, response metadata, and timing
The policy decision that allowed the tool call (for example, "allowed by rule X")
The final outcome and any side effects (records created, messages sent)

2) Over-broad tool permissions

Amit warned: "One poorly constrained tool call and you're explaining why your agent DDoS'd a partner API." That example is not hypothetical. If an agent retries aggressively, parallelizes across sub-agents, or misreads rate limit signals, it can behave like an attacker.

3) Compounding behaviors

Amit wrote: "Without guardrails, agent behaviors compound fast." This is the multi-agent amplifier effect. One agent produces an imperfect plan, another agent executes it literally, and a third agent tries to correct the mess by taking even more actions. If each step lacks constraints, the system can spiral.

Key insight: Multi-agent systems can turn small ambiguity into large action.

What works in production (and why)

Amit listed five practices that show up in mature deployments. I will expand each into concrete, buildable patterns.

Observable architectures: loggable and auditable decisions

"Every agent decision must be loggable and auditable" is not just about saving prompts. It is about making the system legible.

Practical tips:

Use structured events, not only raw text. Log: agent, intent label, policy checks, tool call, and result.
Store a "reason trace" that explains why a route was chosen (for example, retrieval hit IDs, confidence score, policy rule ID).
Make traces queryable by business identifiers (customer ID, ticket ID, invoice ID), not only by run ID.

If regulators, security, or a customer asks "why did the system do this," you need an answer that is more than "the model decided."

Tool-use controls: least privilege plus approvals

"Scope permissions tightly" is the principle, but execution is where teams stumble.

A workable approach is tiered tooling:

Read-only tools by default (search, retrieval, analytics)
Low-risk write tools with tight constraints (create draft, stage changes)
High-risk tools gated by approvals (send email, issue refunds, deploy code, modify permissions)

Add policy checks before every tool call:

Who is the user and what are they allowed to do?
What environment is this (dev, staging, production)?
Does this action exceed thresholds (amount, volume, recipients)?

Approval workflows do not need to slow everything down. Many teams implement "human confirmation only when risk score exceeds N" so routine work stays fast while dangerous actions pause.

Circuit breakers: auto-halt on abnormal behavior

Amit's "auto-halt when behaviors exceed expected parameters" is one of the most underused controls.

Examples of circuit breakers that pay for themselves:

Rate limit breaker: stop if tool calls per minute exceed a bound
Cost breaker: stop if tokens or API spend exceed a budget per run
Blast radius breaker: stop if the agent attempts to modify more than X records
External dependency breaker: stop if error rate spikes or latency grows

The key is to treat "halt" as a safe state, not a failure. A halted run should produce a clear incident record and a handoff to a human.

Human-in-the-loop: escalate, do not assume

Amit wrote: "Agents should escalate, not assume." In practice, that means designing escalation as a first-class path.

Good escalation design:

The agent summarizes context, intent, and proposed action
It lists uncertainties and what it needs from the human
It provides one-click options (approve, edit, reject, request more info)

This reduces cognitive load and prevents rubber-stamping.

Governance from day one: security is table stakes

"Security isn't a bolt-on" is the right mindset, because agentic systems expand your attack surface:

Prompt injection via web pages, emails, documents
Data exfiltration through tool outputs
Indirect prompt injection through retrieval content

Treat governance as part of the architecture:

Threat model the agent and each tool
Red-team with adversarial inputs and tool misuse scenarios
Maintain allowlists for domains, actions, and data fields
Separate duties: do not let the same agent both decide and execute high-risk actions without checks

A production readiness checklist (in Amit's spirit)

Amit asked three questions: "Can you explain every decision your agents make? Do you have rollback mechanisms? Can you halt unexpected behaviors in real time?"

Here is a slightly expanded checklist you can use before you ship:

Explainability: Can you reconstruct the chain of decisions and tool calls for any run?
Containment: What is the maximum damage a single run can do (records, dollars, messages)?
Rollback: For each write action, do you have an undo path or compensating transaction?
Real-time control: Can you kill a run instantly and block specific tools during an incident?
Monitoring: Do you alert on abnormal tool usage patterns, not just errors?
Ownership: Who is on call, who approves policy changes, and who signs off on tool permissions?

Key insight: If you cannot stop it, you do not control it.

Closing thought: powerful autonomy needs boring discipline

Amit Lulla's post is a reminder that the most important work in agentic AI is often "boring" engineering: logs, permissions, thresholds, approvals, and incident playbooks. That discipline is what turns autonomy from a demo into a dependable product.

If you are building agentic AI, take Amit's challenge seriously. Governance is not a tax on innovation. It is what makes innovation safe to scale.

This blog post expands on a viral LinkedIn post by Amit Lulla. View the original LinkedIn post →