Human in the loop decision in healthcare advertising. Preventing bad healthcare adverts from being all over the place. Powered by Proof of Review.
Chapter 1 · the person
It is 11 p.m. Someone whose hair is growing back after chemotherapy opens their phone.
11:02●●● ▮
This advert was written to frighten them. It is one of twelve in our library. Somebody was supposed to check it before it ran, with four hundred more waiting behind it.
Chapter 2 · the rule that exists for them
There is a law for exactly this.
Every rule in this system was copied word for word from the official text and checked against the source by Dr Honey (PHD). Nothing was paraphrased, nothing was learned by a model.
Chapter 3 · the reviewer
The person meant to catch it is drowning.
A reviewer at a platform or an agency sees hundreds of adverts a day. Give them a machine that says "fine" and the research is clear: they check less, not more, and they feel more certain while doing it. Oversight decays exactly when it is relied on.
The case that human oversight degrades under automation, and why healthcare is where it hurts most: Leum and Mukul.
Chapter 4 · what the machine actually does
Watch it check this advert. Every step, nothing hidden.
Meet Sorin. Sorin is the AI in AOTP: it reads the words, reads the picture, and speaks. That is the whole job. Sorin never sees the law and never decides; plain code does, against a dated rulebook.
For the AI experts in the room, in one line: every verdict is a deterministic rule path over verbatim, hash-sealed, human-verified provisions retrieved at run time, with the model confined to quoting the artefact; so each answer is reproducible, citable, dated and attributable, and human oversight is a signed, tamper-evident event rather than a claim.
Someone in the room from another market? Pick theirs; the same five steps run against their law.
Sorin reads the words. With Sorin asleep, the default, the built-in reader does this on this laptop and nothing leaves it. Switched on, it is the AI's only job: it is sent the words, quotes the claims it sees, and never sees the law or gives an opinion.
Keep only what it can quote. A claim that is not on the advert word for word is thrown away, and the throw-away is recorded.
Look the rules up. Not an API, not a model: a file on this machine holding 51 rules, each hashed so it cannot drift. A word search picks the ones that speak to these claims.
Apply them. Plain code: does a rule that applies prohibit the claim, or require something that is missing?
Answer, with the rule quoted, and seal the entry. No rule found means no answer, out loud. The machine cannot say "fine" without a citation. The entry is sealed together with the one before it.
Chapter 5 · the human decides
The machine never approves anything on its own. Here is how the person does.
Green means "no rule we hold is broken". Red means "this rule, quoted". Amber means "a person must decide". Purple means "we hold no rule for this, and we will not guess".
Then the reviewer decides, on the same screen: approve, do not run, or send to a person with the local rulebook. They sign under a reviewer code, never a name, and pick a reason from a fixed list, because free text is where secrets leak. Publishing a verdict is an amber action in the mandate, so the signature issues a token, and the token is itself a sealed entry naming who signed.
Disagreeing with the machine is allowed and is written down as an override. One thing nobody can do: approve a claim that a cited rule prohibits outright. The words have to change. And a refusal to sign is recorded as carefully as a signature.
Chapter 6 · six months later
A complaint arrives. What was checked?
Every check leaves a sealed entry: which rules, which version of the law, who read the advert, who decided. Each entry is sealed together with the one before it, so nobody can quietly rewrite last month. The advert itself is never in it: the record keeps a salted fingerprint, and only a batched fingerprint of many entries is ever published. The one thing that leaves this machine is the advert's words, sent to the AI model to be read, and only when AI reading is switched on.
What is recorded, what never is, and the audit that checks the claim: Dominic.
Chapter 7 · why it is built this way
Governance, assurance, security. One line each.
Governance. The mandate is code, not policy. A forbidden action is refused by a checker, and the refusal writes a receipt.
Assurance. Any verdict can be recomputed from its receipt: same seed, same rulebook fingerprint, same rules, same answer.
Security. The record is chained and anchored outside the operator's control, so the party with the motive to rewrite it cannot.
We wrote every word on this screen so a compliance officer with no AI background can check our work. If they cannot read it, they cannot oversee it, and the human in the loop is decoration.
Chapter 8 · the question we cannot answer yet, and the next phase
What happens when the machine is wrong?
That is the study. In the game, the machine is wrong on purpose one round in five. Does showing a reviewer what the machine checked make them catch it, or just make them more confident? The number on the leaderboard is the answer, and it moves every time someone plays.
What if the advert is a voice, or a video? That is the next phase of this research. Today AOTP reads words and pictures. A spoken advert needs speech turned into words with timestamps; a video needs its frames read and its on-screen text quoted; and then the same gate applies: quote it or drop it, cite the rule or refuse. The people and skills that phase needs are written down in the plan, and the study will run again, because every new kind of advert re-opens the question of whether the reviewer still checks.
Here is a week. Not a product tour. Five days at the ACM Europe School on Responsible AI, Aston, 14 to 18 September 2026, in which a question asked of a speaker became a proof on a page the same night, and two strangers played a game that measured us back.
games by real people
…
opened the sealed record
…
followed the AI when it was wrong on purpose
…
safety property proved
…
The four numbers above are read live from this copy's record and board every time this page opens. They move when someone plays.
Monday 14 · the promise
The AI proposes. The human decides.
Seven words on the first page of the rules file, before any code. A scorer that must cite a rule or refuse. A rulebook copied word for word from seven registers, sealed with one fingerprint. A record where every entry is sealed to the one before it. And one research question: when a machine advises a reviewer, does showing what it checked make the reviewer better, or only more certain?
Tuesday 15 · the corrections
A statistician said "that is not a confidence."
He was right. What we compute is how much of the advert we could check and how much of that was quoted word for word. That is evidence, not probability. So the score is now called what it is, the evidence score, its formula is printed beside every check, and a low score sends the advert to a person. The same day a reviewer said "do not call it explainable AI; you are not explaining a model." Also right. No model made the decision. Sorin reads words off an advert; plain code compares them to a dated rule and prints the rule.
Wednesday 16 · the proof
A question in the morning. A proof by night.
Dr Luca Arnaboldi showed a triage network that was 96 % accurate and still failed a clinical hard limit, found by a verifier that checks every input in a region, not a sample. Our table asked: can the rule "show evidence before you answer" be written down and checked? By the evening it was. The scorer is plain code over a finite vocabulary, so the region can be enumerated: every market, every set of up to three claims, 1,380 points, five invariants, zero counterexamples, re-run on every build and shown live on the Trust page. Luca proves a property of the network. We took the network out of the decision and proved the code that remains.
Wednesday 16, afternoon · the strangers
Two people played. Neither opened the record.
A domain expert, ruling across seven markets they do not work in daily, got four of nine right. Not for lack of skill: nobody holds seven rulebooks in their head. That is the gap the human in the loop is expected to cover today. A second player, shown the AI's advice, got two of five, and on the round where the machine was wrong on purpose, followed it. Nobody clicked the sealed record. This is a demonstration, not a study: two people, no control. The four-condition version is the design. The number that moves on the board when a stranger plays is the finding.
Thursday 17 · people
The kit is finished. The people are not.
Every screen a stranger meets is one click, every number on it is live, and the page passes an accessibility scan on seven screens with nothing failing. What is not finished is what code cannot do: players at the table, a second labeller grading Sorin's reads, and nineteen rules a person must open on the register because the automatic watch could not confirm them. The build froze at 16:00. From here only the record grows.
Friday 18 · the room
Five adverts. One volunteer. A running score.
The demonstration is the game: five rounds, a panel member or a volunteer, never a teammate. Their answer, their certainty, the system's verdict, then the law, quoted, with the link. The score stays on screen. When the machine is wrong on purpose, everyone in the room sees whether the volunteer catches it. Then the sentence we came to say: we started at governance and security. Two talks moved us. The law will not let us remove the human, so the question stopped being "can we replace this control" and became "is this control real, and what would make it real." We are reporting the move, not hiding it.
Say precisely what it shows, no more. One expert, nine items, no control, is illustrative. What is verified: 242 automated tests on every change, the safety property proved over 1,380 points, 22 deliberate breaks with zero crashes, a 27-step browser walk through every screen, and an accessibility scan with zero violations. What is not: real participants at scale, an independent audit of the rulebook by a second lawyer, and the nineteen registers.
Responsible AI, shown rather than claimed. This page is generated from the code and checked live on this machine. A blue line is enforced in code. An amber line has the mechanism but is missing a paper or a person. A grey line is an honest gap. Nothing on this page is legal advice.
Responsible AI, by construction · why it cannot hallucinate a verdict
Nothing is trained on the law. A rule inside a model cannot be cited or dated. The law is looked up in a sealed, dated copy.
Sorin quotes; it never composes a verdict, a rule or a record. The AI proposes claims as exact quotes from the advert; a claim it cannot quote is thrown away and the drop is recorded. On request it may draft a reword of your own advert, and that draft goes through the same check as any advert.
Cite or refuse. No rule found means no verdict, out loud. There is no path from a guess to an approval.
Plain code decides. The verdict is a rule path over retrieved rules, reproducible from its own record.
A person signs. No approval exists without a named-code signature and a reason; overrides are recorded; forbidden claims cannot be approved.
The record holds no subject. Codes and fingerprints only, sealed to the entry before, anchored outside our control.
Checking…
Run the room. Hand the laptop over; every step here is one click. Nobody types a name. The board's third number is Friday's headline.
1 · Three players, four minutes each
Say: "Five adverts. Can each one run in its market? Sometimes the machine lies. Catch it." Then press the next button and hand it over.
Team code aston-am is set for you. Repeat A, C, D for the next three.
2 · The headline number, live
Loading…
"Followed the machine when it was wrong" is the research question. Under 50% by Friday is the story we hope for; whatever it is, it is the truth.
3 · One second labeller, ten minutes
Ideally someone from government. Say: "Good, bad or questionable, and one reason. Twenty-four times. Your agreement with our first labeller shows as you go." Press and hand over.
no second labeller yet
4 · Rehearsal stopwatch
Story with Sorin narrating, Check on the scare advert with one fix, one game round, Trust. Target: under twelve minutes.
0:00
Press again at each hand-off to mark a lap.
5 · The stranger test
Give a person who has never seen this two minutes on the Trust page. Then ask the two questions and write their answer as they said it.
6 · For Dr Honey (PHD) and the deck editor
Nineteen register pages to open, in setup/registers-to-confirm.md. Three deck sentences to change and one row to add, in setup/deck-fixes.md. Both are on paper; neither needs this laptop.
Label the adverts. Twenty-four adverts, one at a time: good, bad or questionable, and one reason from a short list. Under a labeller code, never a name. The frozen answer key is not touched; your labels go to their own file, and your agreement with Dr Honey (PHD) is computed as you go. About ten minutes.
Where can this advert run? Pick one, or bring your own. Every market is checked at once, words and picture. Fix the words, watch the map go green. Then you sign. The machine only proposes.
Where your advert goes the complete list, before you drop anything
Stays on this machineThe check runs here. The page is reachable from this laptop only.
Goes to the AI only if you switch it onThen the advert's words or picture go to the reading service, under contract, with fixed instructions, and come straight back. Never the law, never the record. Sorin rests by default. The service is named on the Trust page, where it belongs.
Never enters the recordThe sealed record holds rule codes, verdicts and a salted fingerprint of the advert. Not one word of it, no brand, no client, no name.
The witness gets a fingerprintOne fingerprint over a batch of entries goes to an outside witness on a fixed clock. Nothing else.
Leaves only when you send it"Copy report" puts the advert's words in your clipboard for the customer. That is your act, not the machine's.
Remembered on disk, to run offlineThe AI's answers are cached on this machine so the same advert is never sent twice. Real adverts need the cache encrypted and expiring; see the compliance map.
The adverts
Drop your own picture here
Advert received
Words read
Claims found
Law checked
Your decision
Sealed & published
Where it can run
What the four colours mean green · red · amber · purple
🟢 Green · can runNo rule we hold is broken.You: sign it off.
🔴 Red · cannot runA rule is broken; it is quoted.You: fix the words.
🟠 Amber · a person decidesA rule needs something only a person can check.You: look, then sign.
🟣 Purple · could not checkNo health claim we recognise, or no rule for this market.You: a local expert looks.
Try a fix edit the words, re-check every market
Change the words and press Check again. Every market is re-checked and the page says what moved. If an advert breaks two rules, fix both.
Report for the customer:Issuing a report is an amber action: it is recorded under your reviewer code with an export token.
GAS board: Governance, Assurance and Security, as a daily habit. Governance is who may do what and who signed. Assurance is the evidence it worked. Security is what can never leak or be forged. This board counts each from the sealed record: fingerprints, decisions and reviewer codes, never an advert's words. Every signed decision counts the same: "Do not run" and "Send to a person" count exactly like "Approve", and disagreeing with the machine counts exactly like agreeing. Nothing here rewards a green result.
Loading the board…
Before an advert runs
A scheduler or publishing tool asks one question: is there a signed decision, for these exact words, in this market, that allows running? It sends the proof number from the customer report and the advert's words to /proof/verify. Changed words, the wrong market, a later "do not run" or a broken record all answer no.
Nothing learns from your adverts, and nothing learns the law. The rulebook is watched, not learned: on a schedule every rule is checked against its register, word for word, and anything unconfirmed is put in front of a person. An AI model only reads the words and points at the claims; it never sees the law and never decides. Plain code looks the law up and applies it. Below is everything it knows: the rules, copied word for word from the official law of each market with a link to read them, and the adverts, with the human's label next to the machine's answer.
Ask our own documents retrieval only, so nothing can be invented
Why nothing here is written by a model. A model that writes an answer can invent one; that is what hallucination is, and in a compliance tool an invented sentence is a liability. So this search never writes. It finds the passage that already exists and shows it with the file and the line, so every answer can be checked against its source. The whole product runs on the same principle: Sorin quotes, it never composes.
The answer is a passage quoted from a document, with its file and line. Nothing is written by a model, so nothing can be made up. Drop more documents in shared/team-docs/ and run ./por docs-index.
How the scores are made · for the technical reviewer
The evidence score on every check says how much of the advert we could actually check. It is not a probability and does not say how likely we are to be right. It is a weighted sum of four parts, each between 0 and 1, computed from that check's own numbers, and it can only lower an answer: a low score sends the advert to a person. Weights are versioned and the version is written into every sealed entry, so a change of formula cannot hide. Source: src/por/confidence.py.
evidence score = 0.30 × coverage + 0.30 × match strength + 0.25 × quoted claims + 0.15 × freshness
coverage = claims the retrieved rules speak to ÷ claims found in the advert
evidence = strongest rule-match score ÷ 8, capped at 1 (8 is treated as a strong match)
extraction = claims kept with a word-for-word quote ÷ (kept + dropped as unquotable)
freshness = 1 − rulebook age in days ÷ 90, never below 0
band: ≥ 0.75 high · 0.55–0.75 moderate · < 0.55 low, and low turns an approval into "a person must decide"
The trust score on the Trust page is a plain average of four areas: live checks (pass ÷ total), EU AI Act, GDPR and the seven markets, where a line enforced in code counts 1, a mechanism missing its paper or person counts ½, and a not-yet counts 0. Source: src/por/trust.py.
The game score is fixed points, listed on the arena's intro, from src/por/arena.py: right 100, streak 25 per step to 5, fast 20, sure-and-right 30, unsure-and-wrong 10, sure-and-wrong −40, caught the machine 150, followed it when wrong −50, read the receipt 15.
The rulebook is sealed. Check it yourself.
Loading…
How a new rule gets in five steps, and why there is no button for it
The law it checks against · 51 rules · 7 markets
The adverts it has · human label vs machine answer
The twelve pictures carry Dr Honey (PHD)'s labels: bad, good or questionable, locked in the answer key. The twelve text adverts are not yet labelled by a person; a second labeller is a known gap, so in the game their truth is the rulebook's own check, and the game says so.
Human decides · AI Act Art. 14Every check recorded · Art. 12Cites the law · Art. 13Says when AI is on · Art. 50No personal data by design · GDPRSealed rulebookNot yet: cache encryption · processor agreement · EU endpoint
· Checks against 51 rules of real law across 7 markets, every one quoted from the official text (library SOURCED-0001). The record keeps only fingerprints of your advert; its words go to the AI model only when AI reading is on.