|
 |
ICYMI: Product for Engineers is now build mode. Same team, same posts. New brand, new focus. Learn more.
People are trusting their agents to do more work without supervision, but how do you decide when to trust them?
Some think the answer depends on how good the model is – i.e. when models get better, you can trust them to do more. But trusting your agents just because the models got smarter is like skipping your seatbelt because you got a nicer car.
The real answer has nothing to do with the model, and everything to do with the task. You need to build a mental model for when to delegate and how much, which you can then adapt.
Here’s a simple guide to maximizing agent autonomy, so you can ship fast without breaking things.
The two factors that set the ceiling
You need ask yourself two questions first:
1. Is it easy to check the agent’s work?
To run safely on their own, agents need immediate feedback when they’re wrong. This is possible for most code with deterministic checks, like unit and integration tests. But subjective tasks, like renaming a parameter for clarity, are harder without human taste and judgment.
2. Is it cheap to undo an agent’s mistake?
Just like in traditional software engineering, if you want to trust an agent run without supervision, you need a guaranteed Ctrl+Z for worst case scenarios. That’s why StampHog, our PR approval agent, routes anything that contains deny-list keywords to a human.
Together, these two factors¹ point you to one of four levels for any task:
Level 0: Agent as assistant. For tasks that are hard to check and hard to undo. Necessary when dealing with tricky and sensitive code.
Level 1: Human-in-the-loop. For tasks that are hard to check and easy to undo. Usually when you need subjective evaluation.
Level 2: Agent delegation. For tasks that are easy to check and hard to undo. The default ceiling for most dev work today.
Level 3: Self-driving mode. For tasks that are easy to check and easy to undo. Everything is accelerating towards this.
You can map these four levels to a very simple decision tree, which you can apply to any task:
 |
This helps you make decisions about how much to delegate to your agents, but you still can engineer your pipeline to enable higher autonomy for any specific task.
Level 0: Agent as assistant
When the work is hard to check + costly to undo
This is the lowest level of agent autonomy. Think asking ChatGPT for advice, or auto-complete in Cursor, just like the good ol’ days of 2024.
But just because it’s old-fashioned doesn’t mean it’s bad; this mode is ideal when dealing with tricky problems in sensitive code surfaces.
For example, when Dylan updated our feature flag engine to support generic property targeting last year, he had to migrate an assumption that was indirectly baked into every feature flag at PostHog.
This would have been hard for agents to check deterministically since it wasn’t possible to grep for. The update also had a huge blast radius since it touched live customer flags, API response shapes, and reason-scoring functions.
How to level up from here
Break the task down. Small pieces make it obvious where delegation is safe or not. Dylan handed off less critical work, like propagating the new targeting logic across our JavaScript, PHP, Ruby, and Flutter SDKs, to agents while doing the riskier core migration by hand.
Level 1: Human-in-the-loop
When the work is hard to check + cheap to undo
This level of agent autonomy is common for tasks that need subjective evaluation since it’s hard to teach agents taste and judgment (for now, at least).
 |
Human-in-the-loop tasks are considered cheap to undo since the code stays in draft mode and won’t get merged until it’s verified by a human. An undo just means kicking off another iteration.
This code readability refactor by Thomas was mostly done by hand, but it's a good example of something an agent wouldn't know how to grade. It was just a few lines that made it easier for humans to understand – adding comments, grouping actions, swapping strings for an enum – and didn't introduce any breaking changes.
How to level up from here
Use LLM-as-judge. This is how most people are building their agentic code review systems. As models continue to improve, more tasks that require human judgment can be checked with LLMs.
Define a scoped, measurable goal. Success metrics or contracts can work as a proxy for a subjective evaluation. For example, you could instruct a system to experiment with landing page copy until a variant reaches 3% conversion.
Write custom skills. This helps agents produce work that reliably fits your standards, conventions, and taste with less steering required. For example, many developers write custom code review skills to follow team-specific standards.
Level 2: Agent delegation
When the work is easy to check + costly to undo
 |
This is the level that most developer tasks are today. An agent writes code that can be tested deterministically, but the final act of merging it is gated behind a final safety check.
When Robbie rewrote our SQL parser in Rust from scratch, he barely read the code since he had a machine oracle to check the work. But since the parser touches every query at PostHog, the agent’s work was gated behind multiple safety checks: shadow mode in production, then a staged cutover.
How to level up from here
Enforce policies and guardrails with code. Most people default to gating Level 2 tasks behind a human (a.k.a., themselves), but this habit turns you into your own bottleneck. Instead, encode as many of your guardrails directly into your pipeline with policies like dry-running by default, scoping credentials, and putting changes behind feature flags.
Level 3: Self-driving mode
When the work is easy to check + cheap to undo
There aren’t very many tasks in this category yet – just smaller ones like dependency bumps, lint fixes, adding test coverage to existing code. But the category is growing quickly, especially with long-running agents, goal-driven loops, and more complex orchestration.
We’re going all-in at PostHog on making self-driving mode a reality for builders. Last month, for example, we launched Scouts – agents that run on a schedule, investigate signals from product data, and draft a PR based on what they find.
How to level up from here
Train domain-specific models. Much of the next wave of tooling will be about improving verification tasks that are difficult for LLMs today. One obvious path is purpose-trained models that know what “good” looks like in a specific domain, which is why we’re training our own AI models at PostHog.
Build expert-level context banks. A lack of agent autonomy is often just due to a context deficit. We learned firsthand from engineering our context layer for the PostHog Wizard that closing this gap with structured and fresh knowledge is one of the highest-leverage things you can build for reliable agents.
Design clear signals for scouts. The bottleneck for long-running agents is going to be whether they know when there’s work worth doing, and how to distinguish valid signals from random noise.
Words by Jina Yoon, who writes these newsletters at Level 0.
🕵️ Have your agents talk to our agents
🔺 More ways to level up
Agentic Autonomy Levels – Addy Osmani
Repositioning PostHog – Charles Cook
How Cerebras Built Its Enterprise Knowledge Base – Isaac Tai, Daniel Kim, Mike Gao
What is a context warehouse? – Lizzie Epton
The new rules of context engineering for Claude 5 models – Thariq Shihipar
Note that scale isn’t a factor for determining agent autonomy levels. People often conflate the two because multi-agent orchestration makes autonomy urgent. But if you get autonomy right at the task level, scale takes care of itself.
| ||||||||||
