DishwasherBench, created by Composo, is a game called Spotless that reproduces the geometry and mechanics of a real Bosch SMV4HAX40G dishwasher. Players place 3D models of dishes, glasses and cutlery into the lower and upper baskets, then run a simulated Eco 50 wash cycle. The simulation uses deterministic rigid‑body physics (via Jolt) for loading and models water jets, droplet impact, runoff and detergent chemistry to determine which items emerge clean. A score is awarded for each item that comes out without residue, providing a straightforward proxy for how well the load was arranged.
This image illustrates the interactive loading mechanic described in the first paragraph, where users place 3D models of dishes into the baskets before running a wash cycle.
This screenshot demonstrates how the wash‑score model detects residue on items, supporting the article’s explanation of scoring for cups, glasses and pots.
The benchmark is organized into seven levels of increasing difficulty, each with saved reference loads from three scripted players (careful, careless, random). An official wash of the largest level takes about five minutes on a MacBook Pro with an M3 Pro chip, while a faster preview tier returns results in roughly a minute. According to the project’s documentation, the score can be trusted for judging whether cups, glasses and pots are placed mouth‑down, but it does not yet reliably indicate whether plates and cutlery are stacked so that they wash clean. The docs list specific failure modes: a glass left mouth up or on its side comes out not clean, whereas plates fallen over or leaning on each other come out clean, which is most probably wrong.
This image corresponds to the article’s discussion of scoring reliability and failure modes, illustrating the feedback users receive after each wash cycle.
This illustrates the saved reference loads mentioned in the second paragraph, which provide baseline performances for careful, careless and random players across seven difficulty levels.
Results so far show that two language‑model agents have been run through an OpenAI‑compatible endpoint, completing six episodes across three levels at the official tier. No human baseline has been collected and the Anthropic API path remains untested. Limitations highlighted in the repository include the fact that only the Eco 50 programme is calibrated, drying, spots and film on glass are not modelled, and items moved by the jets or soil settling on clean pieces are not simulated. The developers view DishwasherBench as a step toward creating repeatable, physics‑grounded environments for evaluating AI spatial reasoning, and they invite contributions to expand the set of test washes, improve the wash‑score model for plates and cutlery, and add support for other operating systems.