Judging 771 websites by hand was never going to happen, so we used AI models as judges. It worked well enough to be useful and badly enough to teach us something. Here is what we would tell anyone about to copy the method.
The setup
Claude scored every site on 30 judged criteria from screenshots and crawl data. Six more criteria, such as speed and accessibility, were measured by script. Then ChatGPT re-scored our site and the top 40 without seeing names, and both models ranked those 41 sites head to head in groups of six.
What went wrong
- The judges disagreed. On absolute scores the two models correlated at only 0.40 (Spearman). On head-to-head rankings they agreed more, at 0.68. If you use one judge, you are using one opinion.
- Absolute scores drift upward. ChatGPT gave most of the top sites 16 to 19 out of 20 on nearly every attribute, which compresses the differences. Head-to-head ranking forces a choice and spreads the field out.
- Our rank changed with the method. Our site was 11th of 41 on Claude’s absolute scores, 3rd on Claude’s head-to-head, and 1st on both of ChatGPT’s. The honest summary is a band, not a number.
- Self-preference is real. One of the judges also helped build our site. Cross-vendor blind judging is the minimum fix, and it is still not proof.
- Stills hide motion. Screenshots under-score animation and micro-interactions for every site.
What went right
The judges found real defects we had missed: counters that showed wrong numbers mid-animation, a “four products” versus “eight apps” contradiction, ten contrast failures, and a placeholder where a screenshot belonged. We fixed them, confirmed the contrast fix with an automated accessibility scan, and re-scored the site: web design rank moved from 11th to 8th of 772 and overall from 80th to 46th.
If you copy this method
- Use at least two judges from different vendors, and hide the names from the second one.
- Prefer head-to-head ranking over absolute scores for the final order. Keep absolute scores for finding specific defects.
- Measure everything you can by script: speed, accessibility, structured data. Use the AI only for what needs judgment.
- Report a band and the disagreement, not a single rank.