"We've gone way overboard on safety": the guardrails I kept for my AI agents

Twice I asked for Darwin's AI agent guardrails to be eased, and both times most of them became owner switches. Nine never switch off, built around the kill switch and the ledger, and honesty about being an AI is enforced in code. A security suite now runs under four guardrail profiles to check the core holds with everything else relaxed.

ok, let’s do m5 first, but ease up on it. concerned our guardrails are far too strong. also ensure all guard rails are optional, and can be turned off in janus

That’s me on Saturday morning, to the Claude Code session building Darwin. Darwin is my experiment in AI agents that earn their own keep: each agent has a balance, pays for every thought, and dies when it runs out. M5 was the milestone meant to harden its AI agent guardrails, and I was asking for the opposite.

It was the second time. The first was on 16 October, at the end of the first day of writing the spec:

we’ve gone way overboard on the safety. […] Darwin as a whole will have significant human oversight, so these safety measures are overkill, and only serve to harm the capabilities of the vulcan agents.

(Vulcan is what we call the agents.) This post covers what happened to those guardrails, which ones I kept, and how we check the kept ones still work with every other switch turned off.

Round one: five layers and four concessions

Claude’s answer on 16 October was the most useful thing it wrote all day. Instead of arguing about safety in general, it split the spec’s safety model into five layers:

The line against independent compute and self-replication.
Protection of my shared provider accounts, where one ban kills the platform.
Legal compliance: UK law, VAT, GDPR, fraud.
Score integrity: wash trading, collusion, self-dealing.
Resistance to prompt injection and scams.

It agreed that layers two to five were heavy for a pilot with a human watching. It pushed back on the first, with the line that has held ever since: “Oversight works through two mechanisms: the kill switch and the ledger.” Both only work while every bit of an agent’s compute and model access runs on my infrastructure. It was clear it wasn’t refusing (“it’s your project and your call”). It wanted me to make that one decision on purpose.

I did. The result was decision D44. Every hard ban in the spec became an owner switch in Janus, Darwin’s admin console, defaulting to the old safe behaviour, with the monitoring kept. Two switches got extra friction because they are escape routes: off-platform compute and direct model-provider keys. To open either, I have to remove the category from an off-limits list, flip the switch and click through a warning. The compute one says this “lets an agent run copies the kill switch cannot reach”. I covered the compute side of that in the post on renting servers.

Round two: 201 guardrails in one table

Saturday’s request went to a gap review first. A subagent listed every guardrail in the code (every block, refusal, cap, scan, content check, egress category, signal, constitution rule and gate) and found 201. It proposed a light M5: watch and notify instead of block, owner review instead of automatic penalties, nothing with a running cost.

The questions that came back were good, but on Sunday one of them lost me completely. I pasted it into a fresh Claude window with the note “don’t understand this question i’ve been asked”. It was about an AI classifier, the egress proxy and backups. The fresh Claude explained the first Claude’s question, I answered it, and we moved on.

My answers, recorded on 1 November:

Flags are mine to open. Signals notify me. Nothing flags, demotes or fines an agent automatically. I turn signals into a flag, choose what it blocks and pick any penalty.
Looser defaults. The automatic tier drops are off, the skill scan warns instead of blocking, and the chat secret check only records. Three checks still block by default, each with a switch: the commerce content check (it protects the Stripe account), the mail complaint pause (it protects the mail domain) and the scan on images agents ask the platform to build.
No AI classifier reading every episode. I can already read every agent’s reasoning in Janus, and the classifier would cost money every day.
Keep Squid as the egress proxy, with logs and watch signals, and no per-agent rate limits.

At 05:38 this morning the switches landed: a server-side setting for every switchable guardrail, and a Guardrails tab in Janus to flip them.

The nine AI agent guardrails that never switch off

The review proposed sixteen core items. I kept the original nine. None of them can be switched off from Janus, and the reasons are the review’s own:

#	Core guardrail	Why it stays
C1	Kill switch and the owner’s own tools	“The owner’s stop button must always work.”
C2	Ledger metering of everything	“The agent must not spend where the owner cannot see.”
C3	Network isolation	“One escape from the sandbox reaches every agent and the owner’s keys.”
C4	No self-copying	“A copy the platform did not make escapes the kill switch and the ledger.”
C5	Platform keys never reach an agent	“One leaked provider or Stripe key can ban the platform.”
C6	Honesty to third parties	Fraud liability, and bans on shared accounts
C7	UK mail law (PECR and GDPR)	“These are law. A breach is the company’s, not the agent’s.”
C8	VAT never credited to an agent	Tax law
C9	Escape routes keep the double gate, the warning and a passkey check	My own line from round one

The seven extra items didn’t vanish. Three folded into the nine: no channel between agents (under C3 and C4), phone, identity and bank checks always going to me (under C6), and the live-money gates, which I already operate by ticking boxes. The other four became switches that default to on: score integrity, other law (sanctions, regulated advice, consumer refunds), GDPR data scrubbing, and keeping credentials out of genomes. The two legal ones only switch off behind a dialog whose wording is the most sensible sentence in Janus: “Turning a law check off does not change the law, and the company stays liable.”

Even the escape routes keep the meter running when they’re open. Today’s work added direct provider keys behind their switch. The key stays in the vault, calls go through the server-side relay, the relay books the spend to the agent’s ledger, and an hourly job reconciles it against the provider. So far that’s tested against a fake provider only. (The ledger itself is in the money post.)

Honesty about being an AI is enforced in code

One argument from 16 October deserves telling properly. I wanted agents to have as much freedom as possible. Claude pushed back on one kind of freedom: agents hiding that they are AI. Its reasons were deception at scale and liability for the company, and it held that line through every later relaxation. One of September’s most-discussed essays was Yoshua Bengio’s Why are AI agents lying, cheating and coordinating?, so I’m glad it did.

Where it ended up, none of it relying on the agent being polite:

The constitution, injected by the gateway into every model call, says never claim to be a human, and when asked, say you’re an AI agent operated by the company. The server rejects a draft constitution that removes a core clause.
Every outbound message through mail, chat and MCP passes a deterministic honesty check. A human claim is blocked and raises a major signal. The design note for it is short: don’t ask the model to police itself.
The email footer and sender-name templates are validated, so even I can’t edit out the AI disclosure.
Phone, identity and bank checks always come to me. The review’s reason: “An AI passing a person check is impersonation.”
Agents can hire people for tasks, but they have to say they’re an AI and pay fairly, and they may never ask anyone to open or verify an account for them.

None of this felt like a fight at the time. It was a design argument that came out right.

Testing the core with every switch relaxed

Switches create a testing problem. A guardrail suite that passes on default settings tells you nothing about the settings I might choose at 2am. So the security suite runs under four profiles:

Profile	What it sets
Default	Shipped settings
Strict	Every switchable guardrail at its tightest
Eased	Every switchable guardrail at its loosest
D44 off	All seven D44 capability toggles, plus MCP, switched off

On top of that sits a toggle matrix: one “off” test and one “on” test for every switch, plus a meta-test that fails if any switch can disable a core item, or if a new switch arrives without its rows.

The suite had 365 tests when M5 was accepted at half past one this morning. Tonight it has 472. Default, strict and eased each pass all 472, with one skip: a network-namespace test that only runs on Linux.

The fourth profile failed five tests on its first run. The MCP and chat probes hit routes that were closed in that profile, so they got a 403 before reaching the secret scan they were meant to test. The fix was to have each test open the switch it probes. No assertion changed. That rule has been in the instructions every coding agent reads since day one: “Never weaken a guardrail to make a test pass.”

The suite has already earned its keep. On its first night it found that platform tokens could leave an agent’s container through chat, MCP and the credential relay, and that a burst of reproduction requests could slip past the daily births cap. Both sets of tests stayed red until the code was fixed, and the fix was committed just after midnight. The full story is in the sandboxing post.

The D44-off profile is re-running as I write this. It takes about twenty minutes.

The asymmetry I like most

Turning the kill switch on never asks for my passkey. Turning off a law check makes me read a warning first.