Simon Willison

<start of post>

A tiny benchmark for agentic HTML parsing

I’ve been building small ‘HTML tools’ for a while, and a lot of them were vibe-coded with the help of Claude, ChatGPT and Gemini. The fun part is that you can get surprisingly far if you treat the model like a fast junior collaborator and you’re disciplined about tests.

This week I tried a slightly different approach: I wrote a minimal harness that takes a chunk of HTML, runs it through a parser, and then checks a set of invariants about the output. The goal is not to “look right”, it’s to be measurably correct.

The rule I’m using is simple: make it impossible to ‘pass’ unless the output is actually correct.

One thing I keep seeing in pull requests is untested AI slop. Submitting that is a dereliction of duty as a software engineer. Your job is to deliver code you have proven to work, not code that feels plausible.

So for this harness, every time the agent proposes a change, it has to run the tests. If it can’t run the tests, it doesn’t get to commit. That one constraint changes the whole dynamic.

I started with a small seed suite (a few dozen cases) and then pointed it at a bigger corpus. It worked on the first run. The second run failed because I’d accidentally allowed a degenerate success condition: the parser could throw away nodes and still satisfy the checks.

That failure was useful, because it forced me to tighten the success criteria instead of tweaking the prompt.

I also stole a trick from the big HTML5 conformance suites: treat weird inputs as a feature, not an edge case. Malformed tags, duplicate attributes, odd encodings, things that only show up on the open web.

At that point the agent started doing the thing I actually wanted: not “write a parser”, but brute-force the space of behaviors until it found the smallest change that made one more test go green.

... and then I tried the same prompt against a local quantized model on my laptop, just to see how far things have come. It’s FAST, and the failure modes are instructive. You get less consistent long-range planning, but you still get a lot of useful local moves if the harness is tight.

The interesting bit is how quickly the agent found the edge cases once the loop was set up. That’s the skill I think people are going to have to learn: designing agentic loops, where the model can try things safely, observe the results, and iterate without you hand-holding every step.

Here’s a detail that surprised me. The evaluations you’d normally run (a couple of canned examples) don’t capture the degradation you’ll see in the wild, because models often recover well from isolated mistakes. If you want reliability, you need a harness that can catch the “looks fine” failures too.

I’m also increasingly convinced that privacy constraints make this harder than most people appreciate. If you can’t inspect the exact failure traces (because you never stored them), you’ll end up chasing ghosts.

If you want to build something with an agent, my current checklist is boring on purpose: define a success condition you can test, give it tools that can’t hurt you in ‘YOLO mode’, and make it run the tests every single time.

Tags: ai , llms , python , html , testing , agents

<end of post>