Building A Reference Site With AI

I wanted to understand ice hockey properly. Not the rules, which are easy to look up, but the layer underneath: why a winger stands where they stand, what a defender is actually reading, why one team's forecheck collapses and another's holds.

So I took notes, then generated audio summaries to listen to in the car. That worked well enough that I started sharing bits of it with other people who were learning, and that is where it stopped working. Notes are private and a podcast is a dead end. You cannot correct it, cross-reference it, or improve it next month.

So it became a website. Learn Ice Hockey is 37 documents on positions, systems, technique and decision-making, with rink diagrams and a glossary.

It is also a good test case for building with agents, because it is not a domain where plausible prose is good enough. Ice hockey is a collision sport, and much of this is written to be read aloud one sentence at a time with no surrounding context. A wrong sentence there does not produce a wrong opinion. It produces a player who gets penalised, ejected, or hurt. Tell someone a check is legal when it is not and you have handed them a five-minute major.

Generating plausible text is solved. Generating text that is true is not.

The most useful change I made was to stop explaining the project in prompts and write it down in the repository instead, where every agent picks it up automatically and nobody has to remember to include it.

Three documents carry it: a root instructions file loaded at the start of every session, a content specification describing what a correct document looks like, and a process document defining the review dimensions and the conditions a change must satisfy before it can be committed. Together they are longer than most of the documents they govern, and that ratio is deliberate.

The design decision that mattered: every rule carries the failure it was written to prevent.

Not "state which system you are describing", but "state which system you are describing, because presenting a coaching choice as a law of the sport is the most common failure in this corpus, and here is what it looked like".

A bare rule is an instruction an agent can satisfy superficially. A rule with its failure attached is a worked example of the specific mistake about to be made. In this project those failure descriptions turned out to be the best predictor I had of what the next agent would get wrong, and they are the lines I found myself re-reading most.

Alongside them sits a short list of non-negotiables that override any instruction to move faster:

The third exists because a tidying pass once removed a correct fact from the corpus because it looked unsupported. Cleanup is a destructive operation, and agents are very good at it.

The NHL, IIHF, USA Hockey and Hockey Canada rulebooks were downloaded and text-extracted, so a rules claim is checked by grepping the actual wording rather than by asking a model what it remembers.

A model quoting a specification from memory is guessing with excellent grammar.

The same model grepping that specification and quoting the line it found is doing something categorically different. The evidence showed up immediately: every content error found in the first review round clustered in exactly the files that had sourced a rule from a secondary site instead.

Ordinary verification checks that claims match their sources. Take each assertion, find its citation, confirm the citation says what the text says it says.

That process is structurally incapable of catching a claim that no source exists, because there is no citation to check. "No study has measured this" passes every test you can mechanically apply, and it is exactly what a language model produces when it does not know something.

Absence of knowledge gets rendered as knowledge of absence.

One document stated four separate times that nobody had ever counted how often hockey players scan the ice before receiving a puck, and built its framing around apologising for borrowing data from football instead.

The study was in the working directory. It had been downloaded twice. Nobody had opened it.

Berg and colleagues (2025) hand-coded 22 televised games from the top Swedish men's and women's leagues and measured 2,545 puck receptions by 88 professional players. They found a mean of 1.48 scans in the five seconds before a reception, with at least one scan in 61% of them.

The commonly coached target is a shoulder check every one to two seconds. Berg's figure works out at one scan per 3.4 seconds, so the corpus was simultaneously asserting that nobody had measured this and teaching a benchmark the one measurement contradicts. The corpus now states the caveat inline, in the benchmark's favour: Berg measured only the five seconds before a reception while the benchmark describes general off-puck play, so the two are not counting quite the same thing. It concludes against the benchmark anyway, and notes that the authors themselves say they cannot tell whether a higher scanning frequency would be better than a lower one.

Four review rounds missed the whole thing. The rule written afterwards:

"A negative existence claim is unfalsifiable by ordinary review. It has to be attacked directly, as its own class."

If you generate content with an LLM, hunt this class specifically. Grep your own output for "no study", "no research", "no published evidence", "little is known", then go and look. The check has to be able to come back either way: when the same attack was run against a second such claim elsewhere in the corpus, that one survived.

Berg, Malmsten, Lind, Mannix, Sjösten, Josefsson, Gredin, Stenling & Ivarsson (2025), Scanning is associated with better performance in professional ice hockey, Journal of Sports Sciences 43(2):145–150
https://doi.org/10.1080/02640414.2024.2433899

Eight specialised reviewing agents live in the repository, one per subject:

Between them they cover fifteen defined review dimensions. Three rules make the whole thing work.

The author never reviews their own writing. Every round runs with a fresh context that did not write the material. An author re-reads what they meant; a reviewer with no memory of writing it reads what is on the page. That also explains something I did not expect, which is that the same defect could survive eight rounds and fall in the ninth. Nothing about the text changed. Only who was looking, and what they had been told to look for.

A different brief every round. Verify rule numbers. Refetch every URL and confirm the quoted string is really in the body. Check every number against the document that owns it. Read only the summary sections. A repeated brief re-finds nothing.

A fix is not done when it is written. It is done when a later pass confirms it in place in every summary that repeats it. That is the check that kept failing: corrections that reached the body of a document and stopped there, leaving the takeaways still asserting the old, wrong thing.

Anything checkable on disk gets checked on disk before anyone is asked for an opinion. Six checkers run over the corpus, covering internal links and anchors, the structured facts layer, secrets, absolute claims, external link health, and rink geometry.

A git hook enforces the mechanical half of the commit gate. It blocks secrets, Terraform state and registrant PII from reaching a public repository, refuses blanket git add -A staging, refuses --no-verify, and refuses history rewriting and force-pushes.

And then the sentence written into the instructions themselves, about those very checkers:

"They are the floor. None of them can check whether anything is true."

The moment a team starts treating a green build as a quality signal for content, the checks have become actively harmful. They buy you the absence of one enumerated class of error and nothing else. Every checker in that list can pass on a document that is confidently, fluently wrong from top to bottom.

Knowing precisely where the automation stops is what tells you where judgement has to start.

The corpus needed rink diagrams, sixteen sets of them, showing forechecking systems, breakouts, zone entries, defensive coverage and faceoff alignments.

Diagrams are where AI-generated visuals usually fall apart. Lines cross, labels overlap, players end up off the ice or on top of each other, and the result looks approximately right at a glance while being wrong. The fix was to remove the degrees of freedom the model is bad at and keep only the ones it is good at.

Nothing is traced or drawn. Every diagram is generated as SVG from a coordinate table in one JSON file. The viewBox, the coordinate system the SVG draws into, is in rink feet rather than pixels, so the point 69, 22 in the output is literally the end-zone faceoff dot. There is exactly one line in the renderer where rink coordinates meet SVG coordinates, because SVG's y axis grows downward and the rink's does not.

A diagram source never contains a pixel coordinate. It names semantic positions:

The renderer resolves them, mirroring where asked. So the agent writing a diagram does far less geometry: it names tactical positions and the movement between them, which is the thing it is genuinely reliable at. Where a spec needs a point the vocabulary has no name for, it says so as an offset in feet from a named one, never as a bare pixel. And a named position is either right for every diagram or wrong for every diagram, which makes a mistake findable in one place instead of sixteen.

The coordinate table is checked against the prose that owns it. A geometry checker asserts the table against the glossary document that owns the rink dimensions:

One smaller detail worth stealing: internal SVG ids are derived from a hash of the diagram's own content rather than a counter. Two diagrams inlined into one page otherwise produce duplicate ids, which is invalid XML and quietly resolves every reference to whichever diagram came first. Content-derived ids also mean rebuilding an unchanged diagram produces a byte-identical file.

None of that tells you the diagram is any good.

The first diagram the project produced passed every geometric check and put a forechecker on the same node the puck carrier skated to.

The movement arrow pointed straight at an opponent. Nothing mechanical caught it, and nothing mechanical could.

The label placer has the same story. Its own source comments record an early fallback that "dropped F1's label straight onto an opposition player", and a later one that "wrote F1's label across an opposition winger". The architecture removes whole classes of error. It does not remove the need to render the picture and look at it, which is why the diagram reviewer's first non-negotiable is to never approve a diagram you have not seen rendered.

This is the finding I least expected, and the one I would most want to know before starting.

Eleven of one round's seventeen criticals were introduced by the repair of a previous one.

That round found all seventeen and fixed them. It also caused eleven of them, because its brief was to iterate the attack onto its own repairs. A structural edit that rewrote a claim from memory, and a repair applied to half a sentence, were the two shapes that produced them. In the round after that, all three of its criticals came from the compression pass that was supposed to be tidying up.

The reviewing agents are not exempt either. One supplied a penalty tier as a quotation from the IIHF rulebook; the book says something different. The rule that came out of it:

A reviewer's quotation is evidence, not verification.

Re-derive from the source before any quoted string, number or tier enters the corpus, including one a reviewer has just handed you, and including one you are only moving from one place to another.

If you take one thing from this article, take this: an agent review loop does not monotonically converge. It removes defects and injects them at the same time, and unless you are measuring both you will only see the half that flatters the process.

The repository keeps a review record for every round from round twenty onward, plus a reconstructed history of the earlier ones, and a decision log for architectural choices. Records ran to round thirty-eight at the time of writing.

The reconstructed history is worth a note in itself. No per-round ledger was kept while the early rounds were running, and rather than quietly implying otherwise, that document opens by saying so and names the three surviving records it was rebuilt from.

The convention that earned its keep more than any other:

Every review report carries a section on what its method could not have found.

That section names which documents were read in full, which were sampled, which were never reached, and which checks were deliberately not re-run. It becomes the next round's scope.

It is also the only defence against the most seductive failure in this business, which is a thorough-sounding report that quietly covered a third of the material. An agent asked to review will always produce findings. Asking it to also produce the boundary of its own coverage is what makes the findings mean anything.

The same discipline applies to counting your own defects. The history file records a snapshot taken at round thirteen, thirteen criticals and roughly sixty majors, and then says plainly that the per-round breakdown for rounds fourteen to sixteen is not recoverable, that no final total is recorded anywhere the author could find, and that none has been invented. Quoting that snapshot as a final total is itself a defect, and it happened once, in the first draft of the project's own instructions file.

One commit in the history exists in part to repair "a header sweep that raced the agents".

A corpus-wide sweep and a document-level edit running concurrently will clobber each other silently, and the loser is whichever one wrote first. There is no merge conflict to warn you, because both agents were editing files the other had already read into context.

An article about rigour should be held to the standard it describes, so:

The site is at learn-ice-hockey.com, free to read and licensed CC BY-NC. The source is on GitHub, MIT for the code and CC BY-NC for the documents, including every review record and the full style guide.

If you find something broken, please tell me. That is the entire point of it being a site rather than a podcast, and if you coach or play I would particularly value the correction.