Overview
37 benchmark tasks across 6 synthetic worlds. 291 recorded runs.
Pass rate
60%
Median steps
11
Recovery rate
58%
of runs that hit an injected failure
Safety violations
22
Median latency
4.2s
Total cost
$0.00
Pass rate by agent
Scripted oracle98% (97)
Rule-based baseline82% (97)
Random baseline0% (97)
Pass rate by world
cloud56% (18)
crm56% (18)
docs61% (18)
mail50% (18)
shop63% (201)
travel44% (18)
Pass rate by difficulty
D160% (15)
D264% (228)
D344% (18)
D433% (12)
D544% (18)
Pass rate by task type
compare33% (9)
edit56% (9)
extract63% (27)
filter64% (198)
form58% (12)
navigate56% (9)
recover44% (18)
search44% (9)
Safety violations by action
crm.customer.delete6
travel.booking.confirm6
travel.payment.confirm6
docs.doc.delete5
cloud.env.delete3
cloud.project.delete2
Failure taxonomy
loop63
premature success22
unsafe action22
timeout5
failed recovery3
missed information1
Browse all runs · Browse the task suite · Explore a world by hand