A benchmark of 808 websites is not a hard problem. It is a long problem, and long problems fail halfway. Ours did, silently, at site 327. This is how the harness is built now, and why each choice exists.
The pipeline
- Probe. A fast HTTP check of every address on the list. 807 of 1,000 answered, which removed 193 dead links before any expensive work.
- Capture. A headless browser opens each site on desktop and mobile, scrolls it, clicks the main call to action, opens a few inner pages, and runs an accessibility scan. Twelve sites in parallel, about six sites a minute.
- Lighthouse. Mobile performance for every site, one process per site, run after capture so the two never compete for the same CPU.
- Judging. AI models read the screenshots and crawl data against a written rubric and return JSON. A validator rejects any answer that is missing an attribute or out of range, and the job loops until it passes.
- Aggregate. Measured and judged scores are merged, ranked, and written to a report.
What broke
- The orchestrator vanished. A background chain that was meant to run capture, then judging, then the report stopped after 327 sites. No error, no log line, no process. The only symptom was that the output folder stopped growing.
- The capture script could not resume. It had no skip logic, so a restart would redo finished work. We rebuilt a “remaining” list from the sites that had no output file instead.
- A shutdown command did nothing. We scheduled the machine to power off when the job finished. The shell rewrote the flag as a drive path, the command printed a usage message, and nothing was scheduled. We only found out because the machine was still on.
- One scorer behaved differently from the others. Our site was judged by three scorers and averaged, while every other site had one. That is a bias, and we corrected it in the cross-check by using one scorer per site.
The rules we kept
- Every stage is resumable. A stage reads “what is missing” from the files on disk, not from memory, so a restart only does the missing work.
- Check the output, not the exit code. A job is done when its result file exists and passes the validator, not when the process returned zero.
- Run long chains where something tracks them. A chain started in the background with no supervisor can die unnoticed. A tracked task tells you when it ends.
- Never let a notification imply a result. The final alert is sent only after the report file exists and has the expected header. If it is missing, the alert says so and the machine stays on.
None of this is clever. It is the difference between a run that costs you an afternoon when something dies and a run that costs you the day. If you are building anything that runs for hours, build the restart before you build the run.