Prompt edits are unmeasured by default.
A skill is a document that changes how an agent behaves. Editing one is a behavioural change to production, with none of the machinery a code change would get: no test, no baseline, no way to tell improvement from a change of mood.
Worse, the evidence that motivated the edit is usually a single bad output — the smallest sample size there is.
Current document against one candidate edit.
Each cycle runs the live document and a one-edit candidate against a held-out eval set, measures the difference in pass rate, and gates promotion on the difference exceeding a multiple of the standard deviation — plus a minimum-runs check that fails closed.
One edit at a time is the part that makes it interpretable. A rewrite that changes six things tells you nothing about which of the six mattered.
- The eval set is held out, so the loop cannot optimise against its own feedback.
- A margin smaller than the noise is not an improvement, and is not promoted.
- Too few runs means no verdict, rather than a verdict on thin evidence.
The precheck fails closed.
Before any cycle runs, a precheck decides whether it may. Exit zero proceeds; a global off switch exits ten; a paused individual skill exits eleven. Any uncertainty at all — missing file, unreadable configuration, unknown name — exits non-zero.
That inversion is the whole safety property. A self-modifying loop that treats an unreadable configuration as permission to continue is a loop that runs exactly when you have lost the ability to control it.
It has never been allowed to apply a change.
The loop runs in dry-run only. It writes a preview of the diff it would make and commits nothing. Arming it is a separate, deliberate, human step, and when armed a promotion will require both the metric gate and an independent reviewer's verdict.
This is the honest status and it is the interesting part. A self-optimising loop is easy to build and hard to trust, and the gap between those two is where it has stayed.
- Dry runs write a preview diff; nothing is committed.
- Arming is a human action gated on a written checklist.
- Even armed, the metric gate alone would not be sufficient.
What it is not allowed to touch.
The scope boundary is written into the tool: improvements inside a verifiable-reward boundary only. It may fix a flaw in how a skill is implemented. It may not touch global policy, architectural decisions, product direction, or what a skill is for.
Those are decisions with no eval set. Handing them to a loop that optimises a number is how you get a number that improves while the thing it measures gets worse.