Open Aya OS — Self-Improvement
Keep the change only if it's better.
Aya proposes a small, single-variable change to her own reasoning prompt, runs it against the current champion on the exact same tasks, and promotes it only when it beats the champion on a single comparable metric — with a regression guard and a cost tie-break. Inspired by Karpathy's autoresearch loop, wired into our own eval engine.
promote bar: +0.02 score · regression cap: 0.5
- Experiments
- 0
- Promotions
- 0
- Champion score
- —
- Best Δscore
- —
Hypothesis leaderboard
Each mutation is a single-variable change to Aya's system prompt, ranked by the best score delta it has earned against the champion on identical inputs.
| # | Hypothesis | Runs | Best Δ | Status |
|---|---|---|---|---|
| 1 | self-consistency Asking for a quick second-pass check before answering reduces careless errors. | 0 | — | — |
| 2 | units-and-edge-cases Explicitly naming units and edge cases catches off-by-one and unit errors. | 0 | — | — |
| 3 | terse-execute Trimming verbose intermediate work keeps the model on-task and lowers cost. | 0 | — | — |
| 4 | restate-constraints Listing the hard constraints up front improves constraint adherence. | 0 | — | — |
Experiment journal
Newest first. Every run is recorded — kept or discarded — with the measured deltas and the reason for the verdict.
Loading journal…
How a winner is decided
- Same inputs. Champion and candidate are scored on the identical seed corpus through the production scoring path — no cherry-picking.
- One metric. Mean score (0..1). A candidate must clear the improvement bar to win, so we never promote on noise.
- Regression guard. Even if the mean rises, a candidate that tanks any single task beyond the cap is rejected.
- Cost tie-break.On a score wash, the cheaper config wins — but only if it's meaningfully cheaper.
- Everything is journaled. Kept or discarded, every experiment leaves a receipt you can read below.