DishwasherBench – Can Frontier Models Stack a Dishwasher?
DishwasherBench turns dishwasher loading into a verifiable benchmark for models and people, but its wash scoring is calibrated on one real test and remains unreliable for plates and cutlery.
DishwasherBench is an early-release benchmark that turns loading a Bosch SMV4HAX40G dishwasher into a deterministic game, Spotless, playable in a browser or by language models through MCP. It simulates rigid-body loading and an Eco 50 wash with GPU-traced water, droplets, run-off and soil chemistry, then scores items for cleanliness.
The project reports two model runs that beat scripted reference loads, but its validation rests on one real test wash and the authors caution that scores reliably judge hollow items placed mouth-down, not plates or cutlery.
Key takeaways
- Built by Composo as version 0.1.0, the benchmark combines a browser game, an MCP server for agents, and a harness that writes replayable episode records.
- Loading uses deterministic rigid-body physics with inputs rounded to 1 mm and 1 degree; the wash traces jets on the GPU and models droplets, run-off, soil and detergent chemistry.
- The official wash tier works on 2 mm blocks and takes about 4.9 minutes for the largest level on an M3 Pro MacBook Pro, using 1–2 GB of memory; the fast tier uses 4 mm blocks.
- Scoring marks an item clean if it carries at most 4 specks and 4 mm² of visible soil; seven levels include saved reference loads for careful, careless and random scripted players.
- Validation includes one real test wash of four egg-yolk plates, published class means from a sister machine, and 48 literature phenomena, of which 17 pass and 21 fail in size or sign.
- Two models played six episodes on three levels and beat the careful reference on full levels, but the project calls one episode per cell an anecdote and notes no human baseline or live Anthropic API run.
What people think
Analysis details
4 comments analyzed. Analyzed .
The sample selects active discussion branches and includes available parent comments. It may omit parts of the full thread. Selected themes do not measure community opinion or how common a view is.
Domestic standards as an AGI test
Commenters joke that matching a spouse's exacting dishwasher-loading standards is the harder benchmark, with poor 'wife mode' scores offered as evidence.
Scoring misses unloading ergonomics
One reply argues loading cannot be objectively wrong, but poor sorting makes unloading harder—a criterion the wash-cleanliness score does not capture.
Simulated environments for robot RL
A commenter sees the physical simulation as a template for training robots via reinforcement learning across many real-world tasks.