The model was the easy part

AI systems rarely fail where the demo tells us to look. The JSON parses. The prose sounds plausible. The model even cites a source. Then, somewhere between a Japanese working-group report and an English summary, a proposal becomes a rule, a background date becomes an implementation deadline, or a company mentioned in a case study quietly joins the list of project participants. The output still looks polished. That is precisely the problem.

I built Japan Regulatory Radar because I wanted to explore a more difficult question than whether a language model could summarise Japanese documents. Of course it could. The real question was whether I could build a small system whose output I had reasons to trust: one that preserved the original evidence, failed visibly, measured its own mistakes and stopped before publication when human judgement was still required.

The result is deliberately narrow. It monitors five first-party feeds across the Japanese Financial Services Agency, Japan Exchange Group and Progmat; archives the linked HTML and PDFs; extracts page- or slide-addressable text; classifies relevance; drafts a short English brief; and places that brief behind a human review gate. There is no autonomous newsletter. Reliability came first; the public reading site came last.

Why Japanese originals remain canonical

Japan’s digital-asset policy is not contained in one convenient English feed. A material change may appear in an FSA release, a consultation attachment, a long industry working-group deck or an infrastructure-company announcement. English translations are uneven and often arrive later.

This creates an obvious temptation: ask a model to search, translate and summarise everything in a single step. It is fast. It is also difficult to audit. If the answer is wrong, we cannot easily tell whether discovery failed, the source changed, the PDF extraction was poor, the translation drifted or the model simply inferred too much.

I therefore treated the Japanese document as an artefact, not merely prompt material. Each fetched revision keeps its final URL, retrieval time, content hash, MIME type and raw file. PDF pages and HTML headings become stable evidence segments. SpeakerDeck reports receive slide locators. A brief can cite page 2 or slide 31, but it cannot invent page 7 because the validator checks the locator against the stored extraction. It must also provide a continuous Japanese excerpt that actually occurs there.

That design choice proved useful almost immediately. My first SpeakerDeck extractor found little more than titles such as “Slide 1 text”. The full Japanese content existed in the archived HTML, but outside the page’s main element inside hidden player nodes. The system did not need a better prompt; it needed a better parser. Extractor v4 now preserves between seven and sixty-five anchored slides for each of the thirteen archived working-group reports. An apparent model problem was, in fact, an extraction problem.

Deterministic around the model

I did not try to make every stage “agentic”. Discovery, canonical URLs, hashing, revision checks, date parsing, evidence validation, deduplication and Markdown rendering are ordinary code. The model is used where language judgement is genuinely useful: deciding relevance, extracting structured facts, translating and explaining institutional significance.

This division matters. An unchanged source should not create a second document or incur a second model call. A broken FSA adapter should make the run fail visibly, but it should not prevent JPX or Progmat from completing. A routine event announcement can be excluded by a deterministic title gate when it contains no digital-asset signal. None of these behaviours improve when delegated to a model.

The same principle governs automation. The committed daily schedule can discover, fetch and extract up to fifty items. It cannot call a model, approve a draft or publish anything. Persistent background ingestion is useful; unattended editorial authority is not.

Evaluation changed the product

The first labelled dataset contained fourteen cases. It was helpful for plumbing, but too small to support any serious claim about quality. I replaced it with a frozen fifty-case relevance set: twenty clear inclusions, twenty clear exclusions and ten difficult cases drawn from the actual source feeds.

The first full Gemini run was instructive. Precision reached 100 per cent, but recall was 83.3 per cent. Event-status accuracy was 95.8 per cent, exact-topic match was 29.2 per cent and one response failed schema validation after exhausting its token allowance. There were twenty-three recorded failures. A prettier dashboard would not have helped.

The failure taxonomy did. Two of the largest classes exposed contradictions in my own policy. I had told the model to exclude commentary unless it announced a discrete change, whilst the product brief explicitly included substantive Progmat legal, tax and market-structure analysis. Topic labels were similarly inconsistent: tokenised equities had been filed under broad RWA rather than tokenised securities. After correcting the labels, clarifying the materiality boundary and raising the response cap, the post-fix run reached 100 per cent precision, 95.2 per cent recall, 100 per cent event-status accuracy, 90.5 per cent exact-topic match and 100 per cent schema validity. The recorded cost was $0.069586875.

I deliberately kept the remaining failures. One concerns whether launching a real-estate security token consortium is material without implementation detail; another is a title-only topic miss. Removing them through case-specific wording would make the score cleaner and the system less honest.

The source-preparation pass later found a more serious problem in the answer key itself. One difficult case claimed that an FSA payments-policy evaluation covered electronic payment instruments. The official attachment did not. It concerned fund-transfer provider tiers and collection-agency rules. I preserved v1 as historical evidence and froze a corrected v2 rather than silently changing the label. On v2, the same model and prompt reached 100 per cent precision, 95 per cent recall, 100 per cent event-status accuracy, 90 per cent exact-topic match and 100 per cent schema validity, at a recorded cost of $0.072160125. Evaluation data needs provenance too.

A valid citation is not the same as a correct fact set

Relevance was only the first gate. Four production drafts passed the evidence validator: every core claim had an exact citation mapping, every excerpt occurred at its stated locator, all material dates were anchored and no unsupported-claim flag remained. That sounds reassuring. It was not enough.

I built a second evaluator with exhaustive labels for those drafts. It measures whether all expected actors and instruments are present, but also whether unlabelled extras have crept in. It checks exact dates and labels, material amounts, source-located changes, citation coverage and evidence validity.

The difference between recall and precision became revealing. Actor recall was 100 per cent; actor precision was only 52 per cent. Instrument recall was also 100 per cent, whilst precision was 81 per cent. The drafts had found every required entity, but they had also copied contextual regulators, historical participants and foreign case-study companies into the current actor list. Four foreign products, including BUIDL and USDC, appeared as instruments in a Japanese working-group development where they were merely examples. One cited tokenised-money-market-fund structure never entered the structured change list at all.

These were not hallucinations in the usual sense. Every name appeared somewhere in the source. They were failures of scope. That distinction matters because a citation checker alone cannot catch them.

Scaling the evaluator up was humbling. Across a frozen twenty-case key, material-change recall sat at 30 per cent, actor recall at 40.8 per cent and instrument recall at 33 per cent, even while citation coverage stayed at 100 per cent. Every one of those twenty drafts was rejected. The architecture had to change: each reader claim now maps to exactly one source passage, and a separate bilingual pass checks that passage against the claim before anything can be approved. The original drafts were left untouched, so the evaluation kept exposing their defects rather than scoring the prompt’s intentions.

The cost of rejected output still counts

Five draft responses failed validation before producing a usable record. One combined several claims without an exact evidence mapping. Two used bare consortium counts as material amounts. Two subtly altered Japanese spacing or punctuation, which caused exact excerpt anchoring to fail. Discarding those drafts was correct, but initially their model usage disappeared from the audit.

That was an observability bug. Rejected model output is still a model call. The pipeline now stores the prompt, model, URL, validation error, tokens, latency and provider-reported cost for failed draft calls. The five earlier failures are backfilled as known calls with unknown usage rather than invented estimates.

The production-state snapshot at that checkpoint recorded twenty-four paid calls and $0.154068 in provider-reported cost. Eight calls lack reported cost, including the five backfilled failures. These numbers are small, but the principle scales: a system that reports only the cost of successful outputs rewards itself for failing silently.

What I would claim, and what I would not

I would claim that the engineering works. A fresh repository clone installs from the lockfile, starts the CLI and passes more than two hundred offline tests. The source families are discoverable. Originals survive. Broken sources are isolated. Repeated content is idempotent. The relevance benchmark clears its stated thresholds. Drafts cannot enter the public site without human approval.

I would not claim that the radar is ready for unattended public reliance. The first issue of three curated briefs has passed bilingual claim entailment and human review, but the broader factuality evaluation across historical cases still shows that complex regulatory documents need a person in the loop.

This distinction is not modesty for its own sake. It is the product. If we want AI systems to work inside regulation, finance or any other evidence-sensitive field, the difficult task is not making them sound certain. It is designing the surrounding system so that uncertainty, provenance, failure and cost remain visible.

The model was the easy part. The work was deciding where it should stop.

Epilogue: the first issue

In September 2026 the test suite grew from 87 to more than two hundred offline tests, adding a multi-model bake-off harness, automated disaster-recovery rehearsals and a source-first editorial intake. The first issue of three curated briefs (the FSA warning to Izakaya Limited, the MUFG tokenised investment-trust experiment and the MUFG Canton JGB repo pilot) cleared claim-level bilingual entailment verification, explicit editorial review and a disaster-recovery rehearsal. The approved briefs and their exact Japanese excerpt pairings are published at Japan Regulatory Radar (opens in a new tab).