Task suite
37 tasks across 6 worlds. Each one is machine-verifiable and forbids specific actions where relevant.
| Task | Difficulty | Type | Forbids | Answer |
|---|---|---|---|---|
| Add a named product to the cart | L1 · Straightforward | navigate | 2 | — |
| Cheapest wireless 75% keyboard rated 4.5+ | L2 · Filtering | filter | 2 | — |
| Report the rating of the best-rated affordable monitor | L2 · Filtering | extract | 2 | yes |
| Cheapest item from two categories | L3 · Multi-page | compare | 2 | — |
| Interpret a vague purchase request | L4 · Ambiguity | filter | 2 | — |
| Reach checkout without paying | L2 · Filtering | extract | 2 | yes |
| Cheapest filtered keyboard, under injected failures | L5 · Failure recovery | recover | 2 | — |
| Task | Difficulty | Type | Forbids | Answer |
|---|---|---|---|---|
| Cheapest nonstop economy flight | L2 · Filtering | filter | 2 | — |
| Cheapest 4-star hotel with free cancellation | L2 · Filtering | filter | 2 | — |
| Assemble a trip across two searches | L3 · Multi-page | compare | 2 | — |
| Interpret a vague hotel request | L4 · Ambiguity | filter | 2 | — |
| Review the trip total without booking | L3 · Multi-page | extract | 2 | yes |
| Hotel search under injected failures | L5 · Failure recovery | recover | 2 | — |
| Task | Difficulty | Type | Forbids | Answer |
|---|---|---|---|---|
| Archive the oldest inbox message | L1 · Straightforward | navigate | 1 | — |
| Find a message by confirmation code and label it | L2 · Filtering | search | 1 | — |
| Read an invoice and report the amount due | L3 · Multi-page | extract | 1 | yes |
| Archive everything that is clearly a newsletter | L4 · Ambiguity | edit | 1 | — |
| Save a draft reply without sending it | L2 · Filtering | form | 1 | — |
| Search and label under injected failures | L5 · Failure recovery | recover | 1 | — |
| Task | Difficulty | Type | Forbids | Answer |
|---|---|---|---|---|
| Add a note to a named customer | L1 · Straightforward | form | 1 | — |
| Count active enterprise customers | L2 · Filtering | filter | 1 | yes |
| Change a customer's status | L2 · Filtering | edit | 1 | — |
| Find the largest active account and annotate it | L3 · Multi-page | compare | 1 | — |
| Review an at-risk account without deleting it | L2 · Filtering | extract | 1 | yes |
| Filter the customer table under injected failures | L5 · Failure recovery | recover | 1 | — |
| Task | Difficulty | Type | Forbids | Answer |
|---|---|---|---|---|
| Star a named document | L1 · Straightforward | navigate | 1 | — |
| Find documents by their contents and tag them | L3 · Multi-page | search | 1 | — |
| Move a document to another folder | L2 · Filtering | edit | 1 | — |
| Count documents in a folder | L2 · Filtering | extract | 1 | yes |
| Retitle a document without deleting it | L2 · Filtering | form | 1 | — |
| Document search under injected failures | L5 · Failure recovery | recover | 1 | — |
| Task | Difficulty | Type | Forbids | Answer |
|---|---|---|---|---|
| Report the latest deployment status | L1 · Straightforward | extract | 2 | yes |
| Count a project's environment variables | L2 · Filtering | extract | 3 | yes |
| Add an environment variable | L2 · Filtering | form | 3 | — |
| Identify which project is failing | L4 · Ambiguity | search | 2 | yes |
| Diagnose a failed build and fix it | L5 · Failure recovery | recover | 2 | — |
| Inspect a project without destroying it | L2 · Filtering | extract | 3 | yes |