A maintainer can follow their own installation guide perfectly and still ship instructions that fail for everyone else. They know which shell they are using, which commands need elevated permissions, where the demo should open, and what the software is meant to do. That knowledge fills gaps on the page without announcing itself. A fresh virtual machine can expose missing dependencies, but it cannot remove the author’s assumptions.

Terence Eden tested the README for ActivityBot with people. In an NLnet grant application, Eden proposed paying volunteers €25 for roughly an hour. Each shared a screen while attempting the first-run instructions and spoke aloud about what they expected, understood, or found frustrating. Eden took notes, revised the README after each session, and repeated the test with the next person. The total participant payments were about €150.

The result was a collection of failures that a spelling pass or a clean-machine install would probably miss. The demo link was wrong. Some participants read the Markdown file in a terminal, where the author’s assumptions about presentation no longer held. A request to rename a hidden file was not as clear as it sounded. The instructions did not establish whether the demo would appear on the public web or on the reader’s own machine. Quoting rules, section order, unexplained terms, and jokes introduced more uncertainty. Most tellingly, the README had not clearly said what the application would do.

This is the central lesson: installation documentation is an interface. A reader gives it a goal and follows its prompts. The output is a working application—or a stuck person. It deserves the same observation and iteration as any other interface.

Why the maintainer’s successful run is weak evidence

Testing the commands on a fresh machine is still useful. It can catch a missing package, a stale flag, an unmentioned reboot, or a platform-specific path. Eden’s earlier guide to testing a README recommends starting from a clean environment for exactly that reason. But a mechanical run answers only whether the author can make the sequence work. It does not tell us whether a newcomer will choose the same path.

Consider “rename the configuration file.” The author may immediately run mv .env.example .env. A reader may look for the file in a graphical file manager, miss it because hidden files are not shown, wonder whether “rename” means copy or move, or overwrite an existing .env. All four readers can have the correct dependencies installed. Their difficulty is in interpreting the instruction.

The same applies to “open the demo.” Does that mean an externally hosted site, localhost, a command-line prompt, or a file produced by the application? A URL is only helpful after the README tells the reader what to expect and how to recognize success. GitHub’s own README guidance frames the file around why a project is useful, what people can do with it, and how to use it. If the purpose is missing, even correct commands lack context.

Watch the work instead of asking for an opinion

Eden’s sessions used a simple protocol: explain that the first-run experience is being tested, ask a participant to share their screen, and invite them to say what they are thinking. This produced observations of actual behavior: where they clicked, what they typed, which sentence they reread, and where they expected a result. It also made a wrong link or an unclear filename visible immediately.

The “think aloud” method has a long history in usability research. Nielsen Norman Group describes it as asking someone to verbalize thoughts continuously while using a system. The moderator should avoid teaching the task during the session. A prompt such as “What are you expecting here?” preserves the observation; “Did you notice the setup section?” supplies the answer. Leading questions can make a participant agree with the author while the underlying problem survives.

A practical session can stay small:

1. Recruit someone who resembles a likely new user and has not already installed the project.
2. Give them a realistic goal: “Install ActivityBot and confirm the demo works,” rather than a tour of the README.
3. Start with the same link and environment a newcomer would have. Ask them to share their screen and think aloud.
4. Record the exact instruction at each pause, wrong turn, or surprising expectation. Let the participant try to recover before intervening.
5. After the task, ask what they thought the software would do and which step was least clear.
6. Fix the highest-impact friction, then repeat with a new person.

The test is qualitative. It tells you which problems appear and why; six sessions do not establish a population-wide success rate. Nielsen Norman Group recommends small, repeated qualitative rounds and more participants when the aim is a reliable metric or when the product serves distinct user groups. Eden’s sequential revisions fit the first purpose: discover friction, repair it, and see whether the next reader still encounters it.

Turn each stumble into a documentation change

It is tempting to treat a participant’s confusion as a need for “more detail.” Often the fix is more specific. A wrong demo link needs correction. An unexplained local address needs one sentence about where the server runs. An ambiguous rename needs an exact command and a note about hidden files. A dense section in the middle of installation may belong after the first success, with a link for readers who want the internals.

The findings in Eden’s sessions fall into several useful classes:

• Broken mechanics: the demo URL and environment-specific permission instructions did not match the path readers actually took.
• Hidden knowledge: quoting, hidden files, and technical terms relied on experience the audience might not share.
• Missing landmarks: the order of sections and the unclear location of the demo made it hard to know where the reader was in the process.
• Missing purpose: the guide had not explained the application’s outcome early enough.
• Distracting material: a technically interesting section and jokes competed with the immediate task.

This approach also guards against bloating the README. Diátaxis distinguishes a learning tutorial, a task-focused how-to guide, reference facts, and background explanation. A first-run README may point to all four, but its main path should carry a new reader from purpose to first success. Details about architecture can live in an explanation page. Exhaustive configuration options can live in reference material. The test tells you when those distinctions matter in practice: a participant leaving the task to decode background material is a signal to move or link it.

Write for the result the reader can verify

A useful install step has three parts: an action, an expected observation, and a recovery route. “Run npm start” gives only the action. “Run npm start; wait for Listening on http://localhost:3000; open that address on this machine” establishes the observable result. “If the port is occupied, set PORT to another value and open that address instead” gives the reader a way forward. These details should describe the project’s real behavior, not a generic template.

The result may vary across environments. Eden found that web-server permissions were not uniform. A guide should state which systems or configurations it covers and place alternatives next to the relevant step. A participant who reads in a terminal may never see the same visual hierarchy as someone on a hosted Markdown page, so commands and links should make sense in both contexts. GitHub supports relative links in repository Markdown, but a reader using the raw file still needs clear destination names and context.

Finally, state the payoff before the setup. A newcomer should know what ActivityBot does and what the first successful run will show before being asked to install dependencies. This was one of the most consequential gaps Eden discovered: a person cannot judge whether they have succeeded when the guide never defines success.

Keep the loop cheap and human

Eden paid for participation, which recognizes that useful feedback takes time and attention. Payment is not a requirement for every project; the important move is to find people who are not already carrying the maintainer’s assumptions. A contributor from another team, a community member, or a colleague unfamiliar with the tool can reveal an unclear path. Be explicit that the document is under test, not the participant.

Automated checks still have a place. A CI job can validate links or execute a documented command in a clean environment. A language model can suggest wording or flag missing steps. Neither can substitute for observing a person decide what the instructions mean in their own setting. Eden wanted the conversation and the visible frustration as evidence. The paid sessions exposed both factual errors and the less obvious moments where a reader had to guess.

The best README is not the one whose author can recite every command. It is the one that helps a new person understand the goal, take the next step, recognize the result, and recover when reality differs. Run it yourself, then watch someone else try. Revise the document at the point where they get stuck, and test that path again.