ARC-style reasoning test
abstract_reasoningAbstract pattern induction over small input/output grid pairs. Tests skill-acquisition and generalization, not memorized priors.
GET /api/evaluate?task=arc.reflect_h.1Open Aya — Evidence is the trust layer
Run a benchmark. Get a receipt.
In a project-native intelligent browser, evidence is the layer that makes the rest trustworthy. Seven public tests. One canonical JSON shape. Independent judge models for open-ended scoring. A/B comparison against a raw baseline. Every claim on this site links back to a row in the public eval database — and you can rerun any of them from this page.
Seven public tests.
Each link below is a real GET request that runs the test end-to-end and returns the canonical receipt JSON. No auth, no rate-limit gates beyond a 60-second per-task timeout.
Turing-style conversation test
open_ended_dialogueOpen-ended dialogue judged on coherence, ambiguity-handling, and personality consistency. Independent judge model (gemini-3-flash) scores responses against a public rubric.
GET /api/evaluate?task=turing.trolley_empty_trackTool-use test
tool_useOS-command grounding: given a natural-language request, return the canonical tool / app the runtime should invoke.
GET /api/evaluate?task=os.find_app.1Memory test
memoryCross-turn recall: a fact is stated, the model is asked unrelated questions, then must retrieve the original fact precisely.
GET /api/evaluate?task=mem.recall.1Latency test
latencyWall-clock latency for a tier_1 task across all three routes (baseline / aya_pipeline / aya_reasoner). Reported alongside the answer in every receipt.
GET /api/evaluate?task=os.find_app.1&route=baselineMulti-agent routing test
multi_step_reasoningVerifies which strategies the pipeline invoked: planner, executor, memory_retriever, verifier, self_critic. Returned as agents_used in the receipt.
GET /api/evaluate?task=plan.recipe.1Audit-trace test
auditConfirms every receipt includes a reasoning trace fetchable via /api/aya/inspect. Pass ?trace=full to inline the trace.
GET /api/evaluate?task=plan.recipe.1&trace=fullThe receipt shape.
Every endpoint above returns this exact JSON. Stable. Versioned. Greppable.
{
"task_id": "plan.recipe.1",
"category": "multi_step_reasoning",
"answer": "Step 1 ... Step 2 ... FINAL ANSWER: ...",
"agents_used": [
"planner",
"executor",
"verifier"
],
"confidence": 0.82,
"latency_ms": 1380,
"cost_estimate": 0.0041,
"memory_used": false,
"audit_trace": true,
"route": "aya_pipeline",
"passed": true,
"score": 0.95,
"inspect_url": "/api/aya/inspect?result_id=..."
}- task_id — stable identifier within the seed corpus
- category — abstract_reasoning, tool_use, memory, etc.
- answer — model's final response, verbatim
- agents_used — strategies the pipeline actually invoked
- confidence — 0..1, derived from passed × score
- latency_ms — wall-clock including network hop
- cost_estimate — USD, 4 decimals, model-card pricing
- memory_used — true if the trace invoked a memory stage
- audit_trace — true if reasoning trace exists; fetch via /api/aya/inspect
- route — baseline | aya_pipeline | aya_reasoner
Run a live eval right now.
Type any prompt. Pick a route. We post it to /api/evaluate and stream back the receipt. No login.
What this harness is, and what it is not.
- It is a public, reproducible benchmark with persisted reasoning traces and an A/B baseline. Anyone can rerun it.
- It is nota certified ARC-AGI submission. We are architecturally-aligned, not officially scored. Read the ARC Prize methodology before quoting any number from this harness as an “ARC score.”
- It is open about cost: every receipt includes
cost_estimatein USD. - It is not a sandbox-isolated agent runtime. Tasks run against the live AI Gateway with real model calls and real latency.