Open Aya OS, scored against itself.
Every architectural claim Aya makes lives on this page as a number, with a reproducible run behind it. ARC-style reasoning, Turing-style conversation, functional-OS commands — all A/B-compared against the raw base model. No marketing copy, no vibes. Receipts.
Not enough A/B data yet to draw a verdict. Run more evals from the Eval Harness app and check back. This page revalidates every 5 minutes.
Total measurements
A/B: does the wrapping help?
The same task corpus, run three ways. Baseline is Claude Sonnet 4.6 with a generic system prompt — the control runs the same conversation tier as the pipeline so the A/B delta isolates exactly what the cognitive wrapping adds, not a model upgrade. Aya Pipeline runs anthropic/claude-sonnet-4.6 with the seven-stage Cognitive Spine. Aya Reasoner routes through anthropic/claude-opus-4.6 with extended thinking enabled (10k token budget). If the wrapping is real, you should see lift over the baseline.
| Route | Tasks | Pass rate | Latency | Tokens |
|---|---|---|---|---|
| baseline (Claude Sonnet 4.6) | 0 | — | — | — |
| aya_pipeline (anthropic/claude-sonnet-4.6) | 0 | — | — | — |
| aya_reasoner (anthropic/claude-opus-4.6) | 0 | — | — | — |
Time-horizons
METR HCAST framingPass-rate is a percentage. Time-horizon is the length of task — measured in minutes a competent human would need — that this route can complete with given probability. The 50% horizon is the capability frontier; the 80% horizon is the deployable-quality bar. Per METR's published research, frontier-agent 50%-horizons have been doubling roughly every seven months — so these numbers are publishing a dated point on a public trend, not a static claim.
| Route | 50% horizon | 80% horizon | Scored attempts |
|---|---|---|---|
| baseline (Claude Sonnet 4.6) | — | — | — |
| aya_pipeline (anthropic/claude-sonnet-4.6) | — | — | — |
| aya_reasoner (anthropic/claude-opus-4.6) | — | — | — |
Horizons interpolated linearly on log(human_time) between the highest passing bin and the lowest failing bin per METR's HCAST methodology. Bins with fewer than 3 attempts are excluded from the fit. Em-dash (“—”) means the route hasn't been scored on enough tasks at that probability threshold yet. Source: each row is derived from eval_results joined to eval_tasks.human_time_minutes; the math lives in .
By eval tier
Voice/text commands that should produce an OS effect (open app, start timer, dictate note).
Memory, planning, multi-step reasoning across the assistant's tools and skills.
Few-shot grid puzzles, abstract pattern induction, novel-rule generalization.
Trick questions, false-memory traps, irony, multi-turn coherence — judged by an independent model.
What we measured. What we did not.
The point of receipts is to be honest about scope. Here is what this page can and cannot tell you about Aya right now.
What the receipts prove
- Aya routes anthropic/claude-sonnet-4.6 for conversation and anthropic/claude-opus-4.6 for extended-thinking reasoning; the eval baseline runs Claude Sonnet 4.6 without the cognitive spine as the fixed control.
- Every score has a reproducible run id and a full reasoning trace stored in Supabase.
- A/B comparison against the unwrapped base model is real and ongoing.
- Scoring is automated (exact_match, contains, numeric, grid_match) or judged by an independent google/gemini-3-flash judge with a published rubric.
- Latency and token cost of every step is recorded — including the cost-of-thought premium the wrapping adds.
What the receipts do NOT prove
- The 6-agent Strategy Auction is currently implemented as system-prompt routing across the same base model, not 6 independently trained policies.
- Cross-device persistence ships when production Supabase auth ships; local-first is the default today.
- ARC-AGI ceiling tracks the underlying reasoner (anthropic/claude-opus-4.6 with extended thinking — published low-single-digit performance on ARC-AGI-2). The wrapping does not change that ceiling — it is a UX layer, not a reasoning multiplier.
Run your own evals.
The Eval Harness is a built-in app. Open the OS, search “evals” in Spotlight, pick your tier, and watch reasoning traces stream live. Your runs join this page automatically.
Page revalidates every 5 minutes. Last rendered 2026-08-28 06:00:57 UTC.