Notes on governed AI, retrieval, evaluation, auditability, and system design — the lessons that only surface after you have built and measured something. Longer engineering write-ups also live on the Keystone blog.
A clinical copilot with role-based action guardrails — three protocols, generative UI, four governance outcomes (Serve, Block, Route, Mutate). The governance architecture moved from workplace safety to acute-care medicine with zero structural changes, which is the point: governance that generalizes is governance that was designed, not bolted on. Built at the Boston AI Tinkerers hackathon with Sai Gopal Jarabana.
An agentic incident-response interface where every tool call is authorized by the user's role and written to a tamper-evident audit trail, and the generated UI itself reflects what the user is permitted to do. Solo build for the AI Tinkerers Generative UI Hackathon.
Outlined and being written, grounded in the Keystone build and its published eval artifacts.
The documents almost always exist. What is missing is finding the right one, proving it is the source, and enforcing who was allowed to see it. Reframing “knowledge” as authorization-first retrieval is what turns an impressive demo into something you can actually run in a regulated shop.
A demo is built to succeed; a real eval is built to embarrass the system. Adversarial ACL probes, out-of-scope refusals, and published failing runs moved the architecture more than any feature request did. The failing run is the artifact that matters.
Queued. The argument is settled; the piece is not written yet.
The distance between a governed platform and a good demo is the unglamorous part: shared substrate, enforced authorization, reproducible evaluation, and an audit trail that survives scrutiny. A demo never has to have any of it. That is the entire difference.
A tool-using agent is trustworthy only if you can see what it may do, what it did, and why — as explicit, authorized, logged state transitions. The control plane has to be a first-class object you can inspect, not an emergent property of a prompt.
Severity-tier escalation, per-step validation, compliance logging, confidence-threshold refusal — the contact-center industry shipped these to production long before LLM products rediscovered them. What it takes to port that discipline onto the model, as architecture.