Terence Eden conducted a user‑testing experiment for his ActivityBot project, paying volunteers €25 for an hour to follow the project’s README while thinking aloud, as part of an NLnet grant application aimed at improving documentation quality. He sought real‑user feedback to counter the bias that developers often have when writing their own instructions.
During the testing sessions, participants uncovered specific problems: the link to the demo tool was incorrect, many were unsure what “renaming a file” meant—especially for hidden files—technical terminology needed explanation, the ordering of sections was confusing, and the author’s jokes were actively confusing rather than helpful. Some tried to follow the README in a terminal without realizing additional steps were required, and assumptions about prerequisites such as PHP‑FPM configuration were not evident to newcomers. After each session Eden revised the README before proceeding to the next volunteer, creating an iterative improvement loop.
While Eden acknowledges the README is not yet perfect, he notes it is now demonstrably easier to follow. He contrasts this approach with using LLMs to simulate users, arguing that real people provide unique perspectives, emotional feedback, and the crucial ability to hear frustration in their voices—a signal that automated testing cannot replicate. He advises developers to seek real‑user testing for documentation when possible, even if unpaid, because observing actual users struggle yields insights that simulated testing cannot.